Options useful for reading tables via (.)Ĭonfig Class: .scalaĮnables use of the spark file index implementation for Hudi, that speeds up listing of large tables.Ĭomma separated list of file paths to read within a Hudi table.įor use-cases like DeltaStreamer which reads from Hoodie Incremental table and applies opaque map functions, filters appearing late in the sequence of transformations cannot be automatically pushed down. These configs control the Hudi Spark Datasource, providing ability to define keys/partitioning, pick out the write operation, specify how to merge records or choosing query type to read. Useful for uniformly enforcing repeated configs (like Hive sync or write/index tuning), across your entire data lake. Specify a different configuration directory location by setting the HUDI_CONF_DIR environment variable. By default, Hudi would load the configuration file under /etc/hudi/conf directory. Instead of directly passing configuration settings to every Hudi job, you can also centrally set them in a configurationįile nf. Amazon Web Services Configs: Please fill in the description for Config Group Name: Amazon Web Services Configs.Kafka Connect Configs: These set of configs are used for Kafka Connect Sink Connector for writing Hudi Tables.This can be overridden to a custom class extending HoodieRecordPayload class, on both datasource and WriteClient levels. Hudi provides default implementations such as OverwriteWithLatestAvroPayload which simply update table with the latest/last-written record. Record payloads define how to produce new values to upsert based on incoming new record and stored old record. Record Payload Config: This is the lowest level of customization offered by Hudi.Metrics Configs: These set of configs are used to enable monitoring and reporting of keyHudi stats and metrics.Although Hudi provides sane defaults, from time-time these configs may need to be tweaked to optimize for specific workloads. These configs provide deep control over lower level aspects like file sizing, compression, parallelism, compaction, write schema, cleaning etc. Write Client Configs: Internally, the Hudi datasource uses a RDD based HoodieWriteClient API to actually perform writes to storage.Flink Sql Configs: These configs control the Hudi Flink SQL source/sink connectors, providing ability to define record keys, pick out the write operation, specify how to merge records, enable/disable asynchronous compaction or choosing query type to read.Spark Datasource Configs: These configs control the Hudi Spark Datasource, providing ability to define keys/partitioning, pick out the write operation, specify how to merge records or choosing query type to read.At a high level, you can control behaviour at few levels. This page covers the different ways of configuring your job to write/read Hudi tables.
0 kommentar(er)
