org.apache.kudu.flume.sink

Class RegexpKuduOperationsProducer

java.lang.Object

org.apache.kudu.flume.sink.RegexpKuduOperationsProducer

All Implemented Interfaces:

AutoCloseable, org.apache.flume.conf.Configurable, KuduOperationsProducer

@InterfaceAudience.Public
 @InterfaceStability.Evolving
public class RegexpKuduOperationsProducer
extends Object
implements KuduOperationsProducer

A regular expression serializer that generates one Insert or Upsert per Event by parsing the payload into values using a regular expression. Values are coerced to the proper column types. Example: if the Kudu table has the schema key INT32 name STRING and producer.pattern is '(?<key>\\d+),(?<name>\w+)', then the RegexpKuduOperationsProducer will parse the string |12345,Mike||54321,Todd| into the rows (key=12345, name=Mike) and (key=54321, name=Todd). Note: this class relies on JDK7 named capturing groups, which are documented in Pattern.

Regular Expression Kudu Operations Producer configuration parameters

Property Name	Default	Required?	Description
producer.pattern		Yes	The regular expression used to parse the event body.
producer.charset	utf-8	No	The charset of the event body.
producer.operation	upsert	No	Operation type used to write the event to Kudu. Must be 'insert' or 'upsert'.
producer.skipMissingColumn	false	No	Whether to ignore a column if it has no corresponding capture group, or instead completely abandon the attempt to parse and insert/upsert the row.
producer.skipBadColumnValue	false	No	Whether to omit a column value from the row if its raw value cannot be coerced to the right type, or instead complete abandon the attempt to parse and insert/operation the row.
producer.warnUnmatchedRows	true	No	Whether to warn about payloads that do not match the pattern. If this option is not set, event bodies with no matches will be silently dropped.

See Also:

Pattern

Field Summary

Fields

Modifier and Type	Field and Description
static String	DEFAULT_ENCODING
static String	DEFAULT_OPERATION
static boolean	DEFAULT_SKIP_BAD_COLUMN_VALUE
static boolean	DEFAULT_SKIP_MISSING_COLUMN
static boolean	DEFAULT_WARN_UNMATCHED_ROWS
static String	ENCODING_PROP
static String	OPERATION_PROP
static String	PATTERN_PROP
static String	SKIP_BAD_COLUMN_VALUE_PROP
static String	SKIP_MISSING_COLUMN_PROP
static String	WARN_UNMATCHED_ROWS_PROP

Constructor Summary

Constructors

Constructor and Description
RegexpKuduOperationsProducer()

Method Summary

Modifier and Type	Method and Description
void	close() Cleans up any state.
void	configure(org.apache.flume.Context context)
List<Operation>	getOperations(org.apache.flume.Event event) Returns the operations that should be written to Kudu as a result of this event.
void	initialize(KuduTable table) Initializes the operations producer.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PATTERN_PROP

public static final String PATTERN_PROP

See Also:

Constant Field Values

ENCODING_PROP

public static final String ENCODING_PROP

See Also:

Constant Field Values

DEFAULT_ENCODING

public static final String DEFAULT_ENCODING

See Also:

Constant Field Values

OPERATION_PROP

public static final String OPERATION_PROP

See Also:

Constant Field Values

DEFAULT_OPERATION

public static final String DEFAULT_OPERATION

See Also:

Constant Field Values

SKIP_MISSING_COLUMN_PROP

public static final String SKIP_MISSING_COLUMN_PROP

See Also:

Constant Field Values

DEFAULT_SKIP_MISSING_COLUMN

public static final boolean DEFAULT_SKIP_MISSING_COLUMN

See Also:

Constant Field Values

SKIP_BAD_COLUMN_VALUE_PROP

public static final String SKIP_BAD_COLUMN_VALUE_PROP

See Also:

Constant Field Values

DEFAULT_SKIP_BAD_COLUMN_VALUE

public static final boolean DEFAULT_SKIP_BAD_COLUMN_VALUE

See Also:

Constant Field Values

WARN_UNMATCHED_ROWS_PROP

public static final String WARN_UNMATCHED_ROWS_PROP

See Also:

Constant Field Values

DEFAULT_WARN_UNMATCHED_ROWS

public static final boolean DEFAULT_WARN_UNMATCHED_ROWS

See Also:

Constant Field Values

Constructor Detail

RegexpKuduOperationsProducer

public RegexpKuduOperationsProducer()

Method Detail

configure

public void configure(org.apache.flume.Context context)

Specified by:

configure in interface org.apache.flume.conf.Configurable

initialize

public void initialize(KuduTable table)

Description copied from interface: KuduOperationsProducer

Initializes the operations producer. Called between configure and getOperations.

Specified by:

initialize in interface KuduOperationsProducer

Parameters:

table - the KuduTable used to create Kudu Operation objects

getOperations

public List<Operation> getOperations(org.apache.flume.Event event)
                              throws org.apache.flume.FlumeException

Description copied from interface: KuduOperationsProducer

Returns the operations that should be written to Kudu as a result of this event.

Specified by:

getOperations in interface KuduOperationsProducer

Returns:

List of Operation that should be written to Kudu

Throws:

org.apache.flume.FlumeException

close

public void close()

Description copied from interface: KuduOperationsProducer

Cleans up any state. Called when the sink is stopped.

Specified by:

close in interface AutoCloseable

Specified by:

close in interface KuduOperationsProducer