org.apache.kudu.flume.sink

Class RegexpKuduOperationsProducer

java.lang.Object

org.apache.kudu.flume.sink.RegexpKuduOperationsProducer

All Implemented Interfaces:

AutoCloseable, org.apache.flume.conf.Configurable, KuduOperationsProducer

@InterfaceAudience.Public
 @InterfaceStability.Evolving
public class RegexpKuduOperationsProducer
extends Object
implements KuduOperationsProducer

A regular expression operations producer that generates one or more Kudu Insert or Upsert operations per Flume Event by parsing the event body using a regular expression. Values are coerced to the types of the named columns in the Kudu table.

Example: If the Kudu table has the schema:

 key INT32
 name STRING

and producer.pattern = (?<key>\\d+),(?<name>\\w+) then RegexpKuduOperationsProducer will parse the string:

|12345,Mike||54321,Todd|

into the rows: (key=12345, name=Mike) and (key=54321, name=Todd).

Note: This class relies on JDK7 named capturing groups, which are documented in Pattern. The name of each capturing group must correspond to a column name in the destination Kudu table.

RegexpKuduOperationsProducer Flume Configuration Parameters

Property Name	Default	Required?	Description
producer.pattern		Yes	The regular expression used to parse the event body.
producer.charset	utf-8	No	The character set of the event body.
producer.operation	upsert	No	Operation type used to write the event to Kudu. Must be either insert or upsert.
producer.skipMissingColumn	false	No	@deprecated use producer.missingColumnPolicy What to do if a column in the Kudu table has no corresponding capture group. If set to true, a warning message is logged and the operation is still attempted. If set to false, an exception is thrown and the sink will not process the Event, causing a Flume Channel rollback.
producer.skipBadColumnValue	false	No	@deprecated use producer.badColumnValuePolicy What to do if a value in the pattern match cannot be coerced to the required type. If set to true, a warning message is logged and the operation is still attempted. If set to false, an exception is thrown and the sink will not process the Event, causing a Flume Channel rollback.
producer.warnUnmatchedRows	true	No	@deprecated use producer.unmatchedRowPolicy Whether to log a warning about payloads that do not match the pattern. If set to false, event bodies with no matches will be silently dropped.
producer.missingColumnPolicy	REJECT	No	What to do if a column in the Kudu table has no corresponding capture group. If set to REJECT, an exception is thrown and the sink will not process the Event, causing a Flume Channel rollback. If set to WARN, a warning message is logged and the operation is still produced. If set to IGNORE, the operation is still produced without any log message.
producer.badColumnValuePolicy	REJECT	No	What to do if a value in the pattern match cannot be coerced to the required type. If set to REJECT, an exception is thrown and the sink will not process the Event, causing a Flume Channel rollback. If set to WARN, a warning message is logged and the operation is still produced, but does not include the given column. If set to IGNORE, the operation is still produced, but does not include the given column and does not log any message.
producer.unmatchedRowPolicy	WARN	No	What to do if a payload does not match the pattern. If set to REJECT, an exception is thrown and the sink will not process the Event, causing a Flume Channel rollback. If set to WARN, a warning message is logged and the row is skipped, not producing an operation. If set to IGNORE, the row is skipped without any log message.

See Also:

Pattern

Field Summary

Fields

Modifier and Type	Field and Description
static String	BAD_COLUMN_VALUE_POLICY_PROP
static org.apache.kudu.flume.sink.RegexpKuduOperationsProducer.ParseErrorPolicy	DEFAULT_BAD_COLUMN_VALUE_POLICY
static String	DEFAULT_ENCODING
static org.apache.kudu.flume.sink.RegexpKuduOperationsProducer.ParseErrorPolicy	DEFAULT_MISSING_COLUMN_POLICY
static String	DEFAULT_OPERATION
static boolean	DEFAULT_SKIP_BAD_COLUMN_VALUE Deprecated.
static boolean	DEFAULT_SKIP_MISSING_COLUMN Deprecated.
static org.apache.kudu.flume.sink.RegexpKuduOperationsProducer.ParseErrorPolicy	DEFAULT_UNMATCHED_ROW_POLICY
static boolean	DEFAULT_WARN_UNMATCHED_ROWS Deprecated.
static String	ENCODING_PROP
static String	MISSING_COLUMN_POLICY_PROP
static String	OPERATION_PROP
static String	PATTERN_PROP
static String	SKIP_BAD_COLUMN_VALUE_PROP Deprecated.
static String	SKIP_MISSING_COLUMN_PROP Deprecated.
static String	UNMATCHED_ROW_POLICY_PROP
static String	WARN_UNMATCHED_ROWS_PROP Deprecated.

Constructor Summary

Constructors

Constructor and Description
RegexpKuduOperationsProducer()

Method Summary

Modifier and Type	Method and Description
void	close() Cleans up any state.
void	configure(org.apache.flume.Context context)
List<Operation>	getOperations(org.apache.flume.Event event) Returns the operations that should be written to Kudu as a result of this event.
void	initialize(KuduTable table) Initializes the operations producer.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PATTERN_PROP

public static final String PATTERN_PROP

See Also:

Constant Field Values

ENCODING_PROP

public static final String ENCODING_PROP

See Also:

Constant Field Values

DEFAULT_ENCODING

public static final String DEFAULT_ENCODING

See Also:

Constant Field Values

OPERATION_PROP

public static final String OPERATION_PROP

See Also:

Constant Field Values

DEFAULT_OPERATION

public static final String DEFAULT_OPERATION

See Also:

Constant Field Values

SKIP_MISSING_COLUMN_PROP

@Deprecated
public static final String SKIP_MISSING_COLUMN_PROP

Deprecated.

See Also:

Constant Field Values

DEFAULT_SKIP_MISSING_COLUMN

@Deprecated
public static final boolean DEFAULT_SKIP_MISSING_COLUMN

Deprecated.

See Also:

Constant Field Values

SKIP_BAD_COLUMN_VALUE_PROP

@Deprecated
public static final String SKIP_BAD_COLUMN_VALUE_PROP

Deprecated.

See Also:

Constant Field Values

DEFAULT_SKIP_BAD_COLUMN_VALUE

@Deprecated
public static final boolean DEFAULT_SKIP_BAD_COLUMN_VALUE

Deprecated.

See Also:

Constant Field Values

WARN_UNMATCHED_ROWS_PROP

@Deprecated
public static final String WARN_UNMATCHED_ROWS_PROP

Deprecated.

See Also:

Constant Field Values

DEFAULT_WARN_UNMATCHED_ROWS

@Deprecated
public static final boolean DEFAULT_WARN_UNMATCHED_ROWS

Deprecated.

See Also:

Constant Field Values

MISSING_COLUMN_POLICY_PROP

public static final String MISSING_COLUMN_POLICY_PROP

See Also:

Constant Field Values

DEFAULT_MISSING_COLUMN_POLICY

public static final org.apache.kudu.flume.sink.RegexpKuduOperationsProducer.ParseErrorPolicy DEFAULT_MISSING_COLUMN_POLICY

BAD_COLUMN_VALUE_POLICY_PROP

public static final String BAD_COLUMN_VALUE_POLICY_PROP

See Also:

Constant Field Values

DEFAULT_BAD_COLUMN_VALUE_POLICY

public static final org.apache.kudu.flume.sink.RegexpKuduOperationsProducer.ParseErrorPolicy DEFAULT_BAD_COLUMN_VALUE_POLICY

UNMATCHED_ROW_POLICY_PROP

public static final String UNMATCHED_ROW_POLICY_PROP

See Also:

Constant Field Values

DEFAULT_UNMATCHED_ROW_POLICY

public static final org.apache.kudu.flume.sink.RegexpKuduOperationsProducer.ParseErrorPolicy DEFAULT_UNMATCHED_ROW_POLICY

Constructor Detail

RegexpKuduOperationsProducer

public RegexpKuduOperationsProducer()

Method Detail

configure

public void configure(org.apache.flume.Context context)

Specified by:

configure in interface org.apache.flume.conf.Configurable

initialize

public void initialize(KuduTable table)

Description copied from interface: KuduOperationsProducer

Initializes the operations producer. Called between configure and getOperations.

Specified by:

initialize in interface KuduOperationsProducer

Parameters:

table - the KuduTable used to create Kudu Operation objects

getOperations

public List<Operation> getOperations(org.apache.flume.Event event)
                              throws org.apache.flume.FlumeException

Description copied from interface: KuduOperationsProducer

Returns the operations that should be written to Kudu as a result of this event.

Specified by:

getOperations in interface KuduOperationsProducer

Parameters:

event - Event to convert to one or more Operations

Returns:

List of Operations that should be written to Kudu

Throws:

org.apache.flume.FlumeException

close

public void close()

Description copied from interface: KuduOperationsProducer

Cleans up any state. Called when the sink is stopped.

Specified by:

close in interface AutoCloseable

Specified by:

close in interface KuduOperationsProducer