org.apache.kudu.flume.sink

Class RegexpKuduOperationsProducer

java.lang.Object

org.apache.kudu.flume.sink.RegexpKuduOperationsProducer

All Implemented Interfaces:

AutoCloseable, org.apache.flume.conf.Configurable, KuduOperationsProducer

@InterfaceAudience.Public
 @InterfaceStability.Evolving
public class RegexpKuduOperationsProducer
extends Object
implements KuduOperationsProducer

A regular expression operations producer that generates one or more Kudu Insert or Upsert operations per Flume Event by parsing the event body using a regular expression. Values are coerced to the types of the named columns in the Kudu table.

Example: If the Kudu table has the schema:

 key INT32
 name STRING

and producer.pattern = (?<key>\\d+),(?<name>\\w+) then RegexpKuduOperationsProducer will parse the string:

|12345,Mike||54321,Todd|

into the rows: (key=12345, name=Mike) and (key=54321, name=Todd).

Note: This class relies on JDK7 named capturing groups, which are documented in Pattern. The name of each capturing group must correspond to a column name in the destination Kudu table.

RegexpKuduOperationsProducer Flume Configuration Parameters

Property Name	Default	Required?	Description
producer.pattern		Yes	The regular expression used to parse the event body.
producer.charset	utf-8	No	The character set of the event body.
producer.operation	upsert	No	Operation type used to write the event to Kudu. Must be either insert or upsert.
producer.skipMissingColumn	false	No	What to do if a column in the Kudu table has no corresponding capture group. If set to true, a warning message is logged and the operation is still attempted. If set to false, an exception is thrown and the sink will not process the Event, causing a Flume Channel rollback.
producer.skipBadColumnValue	false	No	What to do if a value in the pattern match cannot be coerced to the required type. If set to true, a warning message is logged and the operation is still attempted. If set to false, an exception is thrown and the sink will not process the Event, causing a Flume Channel rollback.
producer.warnUnmatchedRows	true	No	Whether to log a warning about payloads that do not match the pattern. If set to false, event bodies with no matches will be silently dropped.

See Also:

Pattern

Field Summary

Fields

Modifier and Type	Field and Description
static String	DEFAULT_ENCODING
static String	DEFAULT_OPERATION
static boolean	DEFAULT_SKIP_BAD_COLUMN_VALUE
static boolean	DEFAULT_SKIP_MISSING_COLUMN
static boolean	DEFAULT_WARN_UNMATCHED_ROWS
static String	ENCODING_PROP
static String	OPERATION_PROP
static String	PATTERN_PROP
static String	SKIP_BAD_COLUMN_VALUE_PROP
static String	SKIP_MISSING_COLUMN_PROP
static String	WARN_UNMATCHED_ROWS_PROP

Constructor Summary

Constructors

Constructor and Description
RegexpKuduOperationsProducer()

Method Summary

Modifier and Type	Method and Description
void	close() Cleans up any state.
void	configure(org.apache.flume.Context context)
List<Operation>	getOperations(org.apache.flume.Event event) Returns the operations that should be written to Kudu as a result of this event.
void	initialize(KuduTable table) Initializes the operations producer.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PATTERN_PROP

public static final String PATTERN_PROP

See Also:

Constant Field Values

ENCODING_PROP

public static final String ENCODING_PROP

See Also:

Constant Field Values

DEFAULT_ENCODING

public static final String DEFAULT_ENCODING

See Also:

Constant Field Values

OPERATION_PROP

public static final String OPERATION_PROP

See Also:

Constant Field Values

DEFAULT_OPERATION

public static final String DEFAULT_OPERATION

See Also:

Constant Field Values

SKIP_MISSING_COLUMN_PROP

public static final String SKIP_MISSING_COLUMN_PROP

See Also:

Constant Field Values

DEFAULT_SKIP_MISSING_COLUMN

public static final boolean DEFAULT_SKIP_MISSING_COLUMN

See Also:

Constant Field Values

SKIP_BAD_COLUMN_VALUE_PROP

public static final String SKIP_BAD_COLUMN_VALUE_PROP

See Also:

Constant Field Values

DEFAULT_SKIP_BAD_COLUMN_VALUE

public static final boolean DEFAULT_SKIP_BAD_COLUMN_VALUE

See Also:

Constant Field Values

WARN_UNMATCHED_ROWS_PROP

public static final String WARN_UNMATCHED_ROWS_PROP

See Also:

Constant Field Values

DEFAULT_WARN_UNMATCHED_ROWS

public static final boolean DEFAULT_WARN_UNMATCHED_ROWS

See Also:

Constant Field Values

Constructor Detail

RegexpKuduOperationsProducer

public RegexpKuduOperationsProducer()

Method Detail

configure

public void configure(org.apache.flume.Context context)

Specified by:

configure in interface org.apache.flume.conf.Configurable

initialize

public void initialize(KuduTable table)

Description copied from interface: KuduOperationsProducer

Initializes the operations producer. Called between configure and getOperations.

Specified by:

initialize in interface KuduOperationsProducer

Parameters:

table - the KuduTable used to create Kudu Operation objects

getOperations

public List<Operation> getOperations(org.apache.flume.Event event)
                              throws org.apache.flume.FlumeException

Description copied from interface: KuduOperationsProducer

Returns the operations that should be written to Kudu as a result of this event.

Specified by:

getOperations in interface KuduOperationsProducer

Parameters:

event - Event to convert to one or more Operations

Returns:

List of Operations that should be written to Kudu

Throws:

org.apache.flume.FlumeException

close

public void close()

Description copied from interface: KuduOperationsProducer

Cleans up any state. Called when the sink is stopped.

Specified by:

close in interface AutoCloseable

Specified by:

close in interface KuduOperationsProducer