Dataflow Templates 2024-06-11-00_RC00

Release week of 2024-06-10

Dataflow Templates 2024-06-04-00_RC01

Release week of 2024-06-03

Dataflow Templates 2024-06-04-00_RC00

Release week of 2024-06-03

Dataflow Templates 2024-05-28-00_RC00

Release Week of 2024-05-28

Note: This release is being rolled out. It may not be in your region yet.

New Templates

N/A

Improvements

[datastream-to-spanner] Tests for additional CUJs (#1556)
[sourcedb-to-spanner] Add new feature (#1577, #1600, #1604)
[googlecloud-to-neo4j] Use built-in batch grouping transform (#1418)
[All Kafka] Add commit offsets (#1597)
[gcs-to-sourcedb] Add Integration test (#1562)
[kafka-to-kafka] Add authentication with SSL (#1555)
[spanner-change-streams-to-sharded-file-sink] Add Reverse replication reader IT (#1575)

Bug Fixes

[sourcedb-to-spanner] Fixed value mapping for blob and year (1578)

Contributors

Aditya Bharadwaj
Akarys Shorabek
Akshara Uke
Anand Inguva
Cloud Teleport
Deep1998
Dippatel98
djagaluru
Florent Biville
Jeffrey Kinard
jian jin
Manit Gupta
sharan-malyala
Sharan Teja M
Shreya Khajanchi
surabhiad
Svetak Sundhar
Tamanna Kakkar
Tanu Sharma
Vardhan Thigle
Vitaly Terentyev

Dataflow Templates 2024-05-21-00_RC00

New Templates

N/A

Improvements

• [BULK] Misc changes across types, logging and options
• [DataStreamToSpanner] - Convert sharded migration ITs to use Avro
• [Avro] - Add support for interval type
• [Plugins] - Add GcsReadBucket type
• [KafkaToBigQuery] - Add support for writing Kafka keys
• [Plugins] - Add support for KAFKA_TOPIC template parameters.
• Add Spanner PR workflow and exclude Spanner from global workflow
• [GcsToSourceDB] - Spanner Reverse replication writer IT
• [Plugins] - Template parameters: update old types to new ones
• [Documentation ] - Add more docs on adding ITs, flex templates, and deprecating parameters
• optimized the processing for lean Spanner write intervals

Bug Fixes

• [BULK MISC FIXES] - Add error counters; Fix null string handlings

Contributors

@ahmedabu98
@aksharauke
@akvelon
@an2x
@damccorm
@darshan-sj
@Deep1998
@Dippatel98
@manitgupta
@RajKumarGupta
@shreyakhajanchi

Dataflow Templates 2024-05-14-00_RC00

Release week of 2024-05-13

Dataflow Templates 2024-05-13-00_RC00

Release Week of 2024-05-13

New Templates

• Kafka to GCS template (#1448)
• Kafka To BigQuery Template (#1464)
• Python UDFs to MongoDB flex templates (#1497)
• Add Python UDF support to GCS to Splunk template (#1494)

Improvements

• [sourcedb-to-spanner] Unified Schema Mappings For Bulk Reader. (#1460)
• [jdbc] JDBC Table Schema Discovery (#1481)
• Upgrade beam version to 2.56.0 (#1504)
• [jdbc] Implementing JDBCIO.RowMapper to map ResultSet to SourceRow. (#1508)
• [bigtable-to-vectorembedding] Add custom App Profile support for Bigtable to Vector Embedding export (#1505)
• [spanner-migration] Add support for the unified custom logical types: JSON, NUMBER and VARCHAR for Spanner Migration Templates (#1509)
• Xlang Dockerfile speed improvements (#1499)
• [spanner-migration] Support UNION field shema types and improve debugging (#1524)
• [UI] Add parentName and parentTriggerValues attributes for templates (#1456)

Bug Fixes

• Parse secret name and password only when using Auth (#1512)

Contributors

Anand Inguva

Archana Gupta

Danny McCormick

Deep1998

Dippatel98

Jeffrey Kinard

Manit Gupta

Ron Gal

Sharan Teja M

Shreya Khajanchi

Supriya Koppa

Vardhan Thigle

Vardhan Vinay Thigle

Yi Hu

anikin

archanagupta03

fozzie15

gauravjainn

hugo-syn

sharan-malyala

surabhiad

vitaly.terentyev

Dataflow Templates 2024-05-08-00_RC00

Release week of 2024-05-06

Dataflow Templates 2024-04-23-00_RC00

Improvements

[YamlTemplate] speed improvements via distroless Xlang image
[JDBC] discover schema from information system tables

Bug Fixes

[Spanner] preserve null and empty arrays correctly

Contributors

Akshara Uke
Cloud Teleport
Damon
Damon Douglas
Danny McCormick
Deep1998
Jack McCluskey
Yi Hu
aksharauke

Dataflow Templates 2024-04-16-00_RC01

Release Week of 2024-04-16

Note: This release is being rolled out. It may not be in your region yet.

Improvements

• Beam version updated to Beam 2.55.1
• Support FLOAT32 type in v1 templates of Spanner
• YAML template performance improvements

Contributors

Aravind Pedapudi
Damon
Damon Douglas
Danny McCormick
Deep1998
Dippatel98
Dmitrii Burlutckii
Jeffrey Kinard
Manit Gupta
Vardhan Thigle
Yi Hu
djagaluru
fozzie15
sharantej
vitaly.terentyev