//nightlies.apache.org/flink/flink-docs-release-2.2/ Recent content in Apache Flink Documentation on Apache Flink Hugo -- gohugo.io en //nightlies.apache.org/flink/flink-docs-release-2.2/docs/libs/cep/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/libs/cep/ FlinkCEP - Complex event processing for Flink # FlinkCEP is the Complex Event Processing (CEP) library implemented on top of Flink. It allows you to detect event patterns in an endless stream of events, giving you the opportunity to get hold of what’s important in your data. This page describes the API calls available in Flink CEP. We start by presenting the Pattern API, which allows you to specify the patterns that you want to detect in your stream, before presenting how you can detect and act upon matching event sequences. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/execution/execution_configuration/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/execution/execution_configuration/ Execution Configuration # The StreamExecutionEnvironment contains the ExecutionConfig which allows to set job specific configuration values for the runtime. To change the defaults that affect all jobs, see Configuration. Java StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); ExecutionConfig executionConfig = env.getConfig(); Python env = StreamExecutionEnvironment.get_execution_environment() execution_config = env.get_config() The following configuration options are available: (the default is bold) setClosureCleanerLevel(). The closure cleaner level is set to ClosureCleanerLevel.RECURSIVE by default. The closure cleaner removes unneeded references to the surrounding class of anonymous functions inside Flink programs. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/try-flink/local_installation/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/try-flink/local_installation/ First steps # Welcome to Flink! :) Flink is designed to process continuous streams of data at a lightning fast pace. This short guide will show you how to download the latest stable version of Flink, install, and run it. You will also run an example Flink job and view it in the web UI. Downloading Flink # Note: Flink is also available as a Docker image. Flink runs on all UNIX-like environments, i. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/formats/overview/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/formats/overview/ Formats # Flink provides a set of table formats that can be used with table connectors. A table format is a storage format that defines how to map binary data onto table columns. Flink supports the following formats: Formats Supported Connectors CSV Apache Kafka, Upsert Kafka, Amazon Kinesis Data Streams, Amazon Kinesis Data Firehose, Filesystem JSON Apache Kafka, Upsert Kafka, Amazon Kinesis Data Streams, Amazon Kinesis Data Firehose, Filesystem, Elasticsearch Apache Avro Apache Kafka, Upsert Kafka, Amazon Kinesis Data Streams, Amazon Kinesis Data Firehose, Filesystem Confluent Avro Apache Kafka, Upsert Kafka Protobuf Apache Kafka Debezium CDC Apache Kafka, Filesystem Canal CDC Apache Kafka, Filesystem Maxwell CDC Apache Kafka, Filesystem OGG CDC Apache Kafka, Filesystem Apache Parquet Filesystem Apache ORC Filesystem Raw Apache Kafka, Upsert Kafka, Amazon Kinesis Data Streams, Amazon Kinesis Data Firehose, Filesystem //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/datastream/intro_to_datastream_api/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/datastream/intro_to_datastream_api/ Intro to the Python DataStream API # DataStream programs in Flink are regular programs that implement transformations on data streams (e.g., filtering, updating state, defining windows, aggregating). The data streams are initially created from various sources (e.g., message queues, socket streams, files). Results are returned via sinks, which may for example write the data to files, or to standard output (for example the command line terminal). Python DataStream API is a Python version of DataStream API which allows Python users could write Python DatStream API jobs. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/models/openai/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/models/openai/ OpenAI # The OpenAI Model Function allows Flink SQL to call OpenAI API for inference tasks. Overview # The function supports calling remote OpenAI model services via Flink SQL for prediction/inference tasks. Currently, the following tasks are supported: Chat Completions: generate a model response from a list of messages comprising a conversation. Embeddings: get a vector representation of a given input that can be easily consumed by machine learning models and algorithms. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/concepts/overview/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/concepts/overview/ Concepts # The Hands-on Training explains the basic concepts of stateful and timely stream processing that underlie Flink’s APIs, and provides examples of how these mechanisms are used in applications. Stateful stream processing is introduced in the context of Data Pipelines & ETL and is further developed in the section on Fault Tolerance. Timely stream processing is introduced in the section on Streaming Analytics. This Concepts in Depth section provides a deeper understanding of how Flink’s architecture and runtime implement these concepts. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/formats/overview/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/formats/overview/ DataStream Formats # Available Formats # Formats define how information is encoded for storage. Currently these formats are supported: Avro Azure Table Hadoop Parquet Text files Back to top //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/overview/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/overview/ DataStream Connectors # Predefined Sources and Sinks # A few basic data sources and sinks are built into Flink and are always available. The predefined data sources include reading from files, directories, and sockets, and ingesting data from collections and iterators. The predefined data sinks support writing to files, to stdout and stderr, and to sockets. Flink Project Connectors # Connectors provide code for interfacing with various third-party systems. Currently these systems are supported as part of the Apache Flink project: //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/overview/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/overview/ Table & SQL Connectors # Flink’s Table API & SQL programs can be connected to other external systems for reading and writing both batch and streaming tables. A table source provides access to data which is stored in external systems (such as a database, key-value store, message queue, or file system). A table sink emits a table to an external storage system. Depending on the type of source and sink, they support different formats such as CSV, Avro, Parquet, or ORC. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/filesystems/overview/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/filesystems/overview/ File Systems # Apache Flink uses file systems to consume and persistently store data, both for the results of applications and for fault tolerance and recovery. These are some of most of the popular file systems, including local, hadoop-compatible, Amazon S3, Aliyun OSS and Azure Blob Storage. The file system used for a particular file is determined by its URI scheme. For example, file:///home/user/text.txt refers to a file in the local file system, while hdfs://namenode:50010/data/user/text. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/ha/overview/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/ha/overview/ High Availability # JobManager High Availability (HA) hardens a Flink cluster against JobManager failures. This feature ensures that a Flink cluster will always continue executing your submitted jobs. JobManager High Availability # The JobManager coordinates every Flink deployment. It is responsible for both scheduling and resource management. By default, there is a single JobManager instance per Flink cluster. This creates a single point of failure (SPOF): if the JobManager crashes, no new programs can be submitted and running programs fail. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/overview/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/overview/ Deployment # Flink is a versatile framework, supporting many different deployment scenarios in a mix and match fashion. Below, we briefly explain the building blocks of a Flink cluster, their purpose and available implementations. If you just want to start Flink locally, we recommend setting up a Standalone Cluster. Overview and Reference Architecture # The figure below shows the building blocks of every Flink cluster. There is always somewhere a client running. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/configuration/overview/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/configuration/overview/ Project Configuration # The guides in this section will show you how to configure your projects via popular build tools (Maven, Gradle), add the necessary dependencies (i.e. connectors and formats, testing), and cover some advanced configuration topics. Every Flink application depends on a set of Flink libraries. At a minimum, the application depends on the Flink APIs and, in addition, on certain connector libraries (i.e. Kafka, Cassandra) and 3rd party dependencies required to the user to develop custom functions to process the data. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/fault-tolerance/serialization/types_serialization/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/fault-tolerance/serialization/types_serialization/ Data Types & Serialization # Apache Flink handles data types and serialization in a unique way, containing its own type descriptors, generic type extraction, and type serialization framework. This document describes the concepts and the rationale behind them. Supported Data Types # Flink places some restrictions on the type of elements that can be in a DataStream. The reason for this is that the system analyzes the types to determine efficient execution strategies. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/operators/overview/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/operators/overview/ Operators # Operators transform one or more DataStreams into a new DataStream. Programs can combine multiple transformations into sophisticated dataflow topologies. This section gives a description of the basic transformations, the effective physical partitioning after applying those as well as insights into Flink’s operator chaining. DataStream Transformations # Map # DataStream → DataStream # Takes one element and produces one element. A map function that doubles the values of the input stream: //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/overview/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/overview/ Flink DataStream API Programming Guide # DataStream programs in Flink are regular programs that implement transformations on data streams (e.g., filtering, updating state, defining windows, aggregating). The data streams are initially created from various sources (e.g., message queues, socket streams, files). Results are returned via sinks, which may for example write the data to files, or to standard output (for example the command line terminal). Flink programs run in a variety of contexts, standalone, or embedded in other programs. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/datastream/operators/overview/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/datastream/operators/overview/ Operators # Operators transform one or more DataStreams into a new DataStream. Programs can combine multiple transformations into sophisticated dataflow topologies. DataStream Transformations # DataStream programs in Flink are regular programs that implement transformations on data streams (e.g., mapping, filtering, reducing). Please see operators for an overview of the available transformations in Python DataStream API. Functions # Transformations accept user-defined functions as input to define the functionality of the transformations. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/overview/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/overview/ Python API # PyFlink is a Python API for Apache Flink that allows you to build scalable batch and streaming workloads, such as real-time data processing pipelines, large-scale exploratory data analysis, Machine Learning (ML) pipelines and ETL processes. If you’re already familiar with Python and libraries such as Pandas, then PyFlink makes it simpler to leverage the full capabilities of the Flink ecosystem. Depending on the level of abstraction you need, there are two different APIs that can be used in PyFlink: //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/table/udfs/overview/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/table/udfs/overview/ User-defined Functions # PyFlink Table API empowers users to do data transformations with Python user-defined functions. Currently, it supports two kinds of Python user-defined functions: the general Python user-defined functions which process data one row at a time and vectorized Python user-defined functions which process data one batch at a time. Bundling UDFs # To run Python UDFs (as well as Pandas UDFs) in any non-local mode, it is strongly recommended bundling your Python UDF definitions using the config option python-files, if your Python UDFs live outside the file where the main() function is defined. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/concepts/overview/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/concepts/overview/ Streaming Concepts # Flink’s Table API and SQL support are unified APIs for batch and stream processing. This means that Table API and SQL queries have the same semantics regardless whether their input is bounded batch input or unbounded stream input. The following pages explain concepts, practical limitations, and stream-specific configuration parameters of Flink’s relational APIs on streaming data. State Management # Table programs that run in streaming mode leverage all capabilities of Flink as a stateful stream processor. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/functions/overview/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/functions/overview/ Functions # Flink Table API & SQL empowers users to do data transformations with functions. Types of Functions # There are two dimensions to classify functions in Flink. One dimension is system (or built-in) functions v.s. catalog functions. System functions have no namespace and can be referenced with just their names. Catalog functions belong to a catalog and database therefore they have catalog and database namespaces, they can be referenced by either fully/partially qualified name (catalog. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/hive-compatibility/hive-dialect/overview/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/hive-compatibility/hive-dialect/overview/ Hive Dialect # Flink allows users to write SQL statements in Hive syntax when Hive dialect is used. By providing compatibility with Hive syntax, we aim to improve the interoperability with Hive and reduce the scenarios when users need to switch between Flink and Hive in order to execute different statements. Use Hive Dialect # Flink currently supports two SQL dialects: default and hive. You need to switch to Hive dialect before you can write in Hive syntax. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/hive-compatibility/hive-dialect/queries/overview/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/hive-compatibility/hive-dialect/queries/overview/ Queries # Description # Hive dialect supports a commonly-used subset of Hive’s DQL. The following lists some parts of HiveQL supported by the Hive dialect. Sort/Cluster/Distributed BY Group By Join Set Operation Lateral View Window Functions Sub-Queries CTE Transform Table Sample Syntax # The following section describes the overall query syntax. The SELECT clause can be part of a query which also includes common table expressions (CTE), set operations, and various other clauses. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/materialized-table/overview/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/materialized-table/overview/ Introduction # Materialized Table is a new table type introduced in Flink SQL, aimed at simplifying both batch and stream data pipelines, providing a consistent development experience. By specifying data freshness and query when creating Materialized Table, the engine automatically derives the schema for the materialized table and creates corresponding data refresh pipeline to achieve the specified freshness. Core Concepts # Materialized Table encompass the following core concepts: Data Freshness, Refresh Mode, Query Definition and Schema. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/overview/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/overview/ Table API & SQL # Apache Flink features two relational APIs - the Table API and SQL - for unified stream and batch processing. The Table API is a language-integrated query API for Java, Scala, and Python that allows the composition of queries from relational operators such as selection, filter, and join in a very intuitive way. Flink’s SQL support is based on Apache Calcite which implements the SQL standard. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql-gateway/overview/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql-gateway/overview/ Introduction # The SQL Gateway is a service that enables multiple clients from the remote to execute SQL in concurrency. It provides an easy way to submit the Flink Job, look up the metadata, and analyze the data online. The SQL Gateway is composed of pluggable endpoints and the SqlGatewayService. The SqlGatewayService is a processor that is reused by the endpoints to handle the requests. The endpoint is an entry point that allows users to connect. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/overview/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/overview/ Queries # SELECT statements and VALUES statements are specified with the sqlQuery() method of the TableEnvironment. The method returns the result of the SELECT statement (or the VALUES statements) as a Table. A Table can be used in subsequent SQL and Table API queries, be converted into a DataStream, or written to a TableSink. SQL and Table API queries can be seamlessly mixed and are holistically optimized and translated into a single program. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/learn-flink/overview/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/learn-flink/overview/ Learn Flink: Hands-On Training # Goals and Scope of this Training # This training presents an introduction to Apache Flink that includes just enough to get you started writing scalable streaming ETL, analytics, and event-driven applications, while leaving out a lot of (ultimately important) details. The focus is on providing straightforward introductions to Flink’s APIs for managing state and time, with the expectation that having mastered these fundamentals, you’ll be much better equipped to pick up the rest of what you need to know from the more detailed reference documentation. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/hive/overview/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/hive/overview/ Apache Hive # Apache Hive has established itself as a focal point of the data warehousing ecosystem. It serves as not only a SQL engine for big data analytics and ETL, but also a data management platform, where data is discovered, defined, and evolved. Flink offers a two-fold integration with Hive. The first is to leverage Hive’s Metastore as a persistent catalog with Flink’s HiveCatalog for storing Flink specific metadata across sessions. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/memory/mem_setup/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/memory/mem_setup/ Set up Flink’s Process Memory # Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components. While the community strives to offer sensible defaults to all configurations, the full breadth of applications that users deploy on Flink means this isn’t always possible. To provide the most production value to our users, Flink allows both high-level and fine-grained tuning of memory allocation within clusters. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/overview/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/overview/ SQL # This page describes the SQL language supported in Flink, including Data Definition Language (DDL), Data Manipulation Language (DML) and Query Language. Flink’s SQL support is based on Apache Calcite which implements the SQL standard. This page lists all the supported statements supported in Flink SQL for now: SELECT (Queries) CREATE TABLE, CATALOG, DATABASE, VIEW, FUNCTION DROP TABLE, DATABASE, VIEW, FUNCTION ALTER TABLE, DATABASE, FUNCTION ANALYZE TABLE INSERT UPDATE DELETE DESCRIBE EXPLAIN USE SHOW LOAD UNLOAD Data Types # Please see the dedicated page about data types. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/fault-tolerance/state_migration/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/fault-tolerance/state_migration/ State TTL Migration Compatibility # Starting with Apache Flink 2.2.0, the system supports seamless enabling or disabling of State Time-to-Live (TTL) for existing state. This enhancement removes prior limitations where a change in TTL configuration could cause a StateMigrationException during restore. Version Overview # Flink Version Change 2.0.0 Introduced TtlAwareSerializer to support TTL/non-TTL serializer compatibility 2.1.0 Added TTL migration support for RocksDBKeyedStateBackend 2.2.0 Added TTL migration support for HeapKeyedStateBackend Full TTL state migration support across all major state backends is available from Flink 2. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/fault-tolerance/state/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/fault-tolerance/state/ Working with State # In this section you will learn about the APIs that Flink provides for writing stateful programs. Please take a look at Stateful Stream Processing to learn about the concepts behind stateful stream processing. Keyed DataStream # If you want to use keyed state, you first need to specify a key on a DataStream that should be used to partition the state (and also the records in the stream themselves). //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/filesystems/common/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/filesystems/common/ Common Configurations # Apache Flink provides several standard configuration settings that work across all file system implementations. Default File System # A default scheme (and authority) is used if paths to files do not explicitly specify a file system scheme (and authority). fs.default-scheme: <default-fs> For example, if the default file system configured as fs.default-scheme: hdfs://localhost:9000/, then a file path of /user/hugo/in.txt is interpreted as hdfs://localhost:9000/user/hugo/in.txt. Connection limiting # You can limit the total number of connections that a file system can concurrently open which is useful when the file system cannot handle a large number of concurrent reads/writes or open connections at the same time. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/common/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/common/ Concepts & Common API # The Table API and SQL are integrated in a joint API. The central concept of this API is a Table which serves as input and output of queries. This document shows the common structure of programs with Table API and SQL queries, how to register a Table, how to query a Table, and how to emit a Table. Structure of Table API and SQL Programs # The following code example shows the common structure of Table API and SQL programs. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/hive-compatibility/hive-dialect/create/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/hive-compatibility/hive-dialect/create/ CREATE Statements # With Hive dialect, the following CREATE statements are supported for now: CREATE DATABASE CREATE TABLE CREATE VIEW CREATE MARCO CREATE FUNCTION CREATE DATABASE # Description # CREATE DATABASE statement is used to create a database with the specified name. Syntax # CREATE (DATABASE|SCHEMA) [IF NOT EXISTS] database_name [COMMENT database_comment] [LOCATION hdfs_path] [WITH DBPROPERTIES (property_name=property_value, ...)]; Examples # CREATE DATABASE db1; CREATE DATABASE IF NOT EXISTS db1 COMMENT 'db1' LOCATION '/user/hive/warehouse/db1' WITH DBPROPERTIES ('name'='example-db'); CREATE TABLE # Description # CREATE TABLE statement is used to define a table in an existing database. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/formats/csv/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/formats/csv/ CSV Format # Format: Serialization Schema Format: Deserialization Schema The CSV format allows to read and write CSV data based on an CSV schema. Currently, the CSV schema is derived from table schema. Dependencies # In order to use the CSV format the following dependencies are required for both projects using a build automation tool (such as Maven or SBT) and SQL Client with SQL JAR bundles. Maven dependency SQL Client &ltdependency&gt &ltgroupId&gtorg. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/debugging/debugging_event_time/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/debugging/debugging_event_time/ Debugging Windows & Event Time # Monitoring Current Event Time # Flink’s event time and watermark support are powerful features for handling out-of-order events. However, it’s harder to understand what exactly is going on because the progress of time is tracked within the system. Low watermarks of each task can be accessed through Flink web interface or metrics system. Each Task in Flink exposes a metric called currentInputWatermark that represents the lowest watermark received by this task. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/concepts/determinism/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/concepts/determinism/ Determinism In Continuous Queries # This article is about: What is determinism? Is all batch processing deterministic? Two examples of batch queries with non-deterministic results Non-determinism in batch processing Determinism in streaming processing Non-determinism in streaming Non-deterministic update in streaming How to eliminate the impact of non-deterministic update in streaming queries 1. What Is Determinism? # Quoting the SQL standard’s description of determinism: ‘an operation is deterministic if that operation assuredly computes identical results when repeated with identical input values’. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/hive-compatibility/hive-dialect/drop/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/hive-compatibility/hive-dialect/drop/ CREATE Statements # With Hive dialect, the following DROP statements are supported for now: DROP DATABASE DROP TABLE DROP VIEW DROP MARCO DROP FUNCTION DROP DATABASE # Description # DROP DATABASE statement is used to drop a database as well as the tables/directories associated with the database. Syntax # DROP (DATABASE|SCHEMA) [IF EXISTS] database_name [RESTRICT|CASCADE]; The use of SCHEMA and DATABASE are interchangeable - they mean the same thing. The default behavior is RESTRICT, where DROP DATABASE will fail if the database is not empty. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/concepts/dynamic_tables/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/concepts/dynamic_tables/ Dynamic Tables # SQL - and the Table API - offer flexible and powerful capabilities for real-time data processing. This page describes how relational concepts elegantly translate to streaming, allowing Flink to achieve the same semantics on unbounded streams. Relational Queries on Data Streams # The following table compares traditional relational algebra and stream processing for input data, execution, and output results. Relational Algebra / SQL Stream Processing Relations (or tables) are bounded (multi-)sets of tuples. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/execution_mode/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/execution_mode/ Execution Mode (Batch/Streaming) # The DataStream API supports different runtime execution modes from which you can choose depending on the requirements of your use case and the characteristics of your job. There is the “classic” execution behavior of the DataStream API, which we call STREAMING execution mode. This should be used for unbounded jobs that require continuous incremental processing and are expected to stay online indefinitely. Additionally, there is a batch-style execution mode that we call BATCH execution mode. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/advanced/external_resources/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/advanced/external_resources/ External Resource Framework # In addition to CPU and memory, many workloads also need some other resources, e.g. GPUs for deep learning. To support external resources, Flink provides an external resource framework. The framework supports requesting various types of resources from the underlying resource management systems (e.g., Kubernetes), and supplies information needed for using these resources to the operators. Different resource types can be supported. You can either leverage built-in plugins provided by Flink (currently only for GPU support), or implement your own plugins for custom resource types. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/guarantees/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/guarantees/ Fault Tolerance Guarantees of Data Sources and Sinks # Flink’s fault tolerance mechanism recovers programs in the presence of failures and continues to execute them. Such failures include machine hardware failures, network failures, transient program failures, etc. Flink can guarantee exactly-once state updates to user-defined state only when the source participates in the snapshotting mechanism. The following table lists the state update guarantees of Flink coupled with the bundled connectors. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/try-flink/datastream/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/try-flink/datastream/ Fraud Detection with the DataStream API # Apache Flink offers a DataStream API for building robust, stateful streaming applications. It provides fine-grained control over state and time, which allows for the implementation of advanced event-driven systems. In this step-by-step guide you’ll learn how to build a stateful streaming application with Flink’s DataStream API. What Are You Building? # Credit card fraud is a growing concern in the digital age. Criminals steal credit card numbers by running scams or hacking into insecure systems. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/event-time/generating_watermarks/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/event-time/generating_watermarks/ Generating Watermarks # In this section you will learn about the APIs that Flink provides for working with event time timestamps and watermarks. For an introduction to event time, processing time, and ingestion time, please refer to the introduction to event time. Introduction to Watermark Strategies # In order to work with event time, Flink needs to know the events timestamps, meaning each element in the stream needs to have its event timestamp assigned. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/gettingstarted/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/gettingstarted/ Getting Started # Flink SQL makes it simple to develop streaming applications using standard SQL. It is easy to learn Flink if you have ever worked with a database or SQL like system by remaining ANSI-SQL 2011 compliant. This tutorial will help you get started quickly with a Flink SQL development environment. Prerequisites # You only need to have basic knowledge of SQL to follow along. No other programming experience is assumed. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/hints/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/hints/ SQL Hints # Batch Streaming SQL hints can be used with SQL statements to alter execution plans. This chapter explains how to use hints to force various approaches. Generally a hint can be used to: Enforce planner: there’s no perfect planner, so it makes sense to implement hints to allow user better control the execution; Append meta data(or statistics): some statistics like “table index for scan” and “skew info of some shuffle keys” are somewhat dynamic for the query, it would be very convenient to config them with hints because our planning metadata from the planner is very often not that accurate; Operator resource constraints: for many cases, we would give a default resource configuration for the execution operators, i. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/hive/hive_catalog/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/hive/hive_catalog/ Hive Catalog # Hive Metastore has evolved into the de facto metadata hub over the years in Hadoop ecosystem. Many companies have a single Hive Metastore service instance in their production to manage all of their metadata, either Hive metadata or non-Hive metadata, as the source of truth. For users who have both Hive and Flink deployments, HiveCatalog enables them to use Hive Metastore to manage Flink’s metadata. For users who have just Flink deployment, HiveCatalog is the only persistent catalog provided out-of-box by Flink. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/installation/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/installation/ Installation # Environment Requirements # Python version (3.9, 3.10, 3.11 or 3.12) is required for PyFlink. Please run the following command to make sure that it meets the requirements: $ python --version # the version printed here must be 3.9, 3.10, 3.11 or 3.12 Environment Setup # Your system may include multiple Python versions, and thus also include multiple Python binary executables. You can run the following ls command to find out what Python binary executables are available in your system: //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/java_compatibility/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/java_compatibility/ Java compatibility # This page lists which Java versions Flink supports and what limitations apply (if any). Java 11 # Support for Java 11 was added in 1.10.0. Untested Flink features # The following Flink features have not been tested with Java 11: Hive connector Hbase 1.x connector Untested language features # Modularized user jars have not been tested. Java 17 # We use Java 17 by default in Flink 2. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/monitoring/checkpoint_monitoring/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/monitoring/checkpoint_monitoring/ Monitoring Checkpointing # Overview # Flink’s web interface provides a tab to monitor the checkpoints of jobs. These stats are also available after the job has terminated. There are four different tabs to display information about your checkpoints: Overview, History, Summary, and Configuration. The following sections will cover all of these in turn. Monitoring # Overview Tab # The overview tabs lists the following statistics. Note that these statistics don’t survive a JobManager loss and are reset if your JobManager fails over. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/resource-providers/standalone/overview/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/resource-providers/standalone/overview/ Standalone # Getting Started # This Getting Started section guides you through the local setup (on one machine, but in separate processes) of a Flink cluster. This can easily be expanded to set up a distributed standalone cluster, which we describe in the reference section. Introduction # The standalone mode is the most barebone way of deploying Flink: The Flink services described in the deployment overview are just launched as processes on the operating system. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream-v2/overview/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream-v2/overview/ Note: DataStream API V2 is a new set of APIs, to gradually replace the original DataStream API. It is currently in the experimental stage and is not fully available for production. Flink DataStream API Programming Guide # DataStream programs in Flink are regular programs that implement transformations on data streams (e.g., filtering, updating state, defining windows, aggregating). The data streams are initially created from various sources (e.g., message queues, socket streams, files). //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql-gateway/rest/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql-gateway/rest/ REST Endpoint # The REST endpoint allows user to connect to SQL Gateway with REST API. Overview of SQL Processing # Open Session # When the client connects to the SQL Gateway, the SQL Gateway creates a Session as the context to store the users-specified information during the interactions between the client and SQL Gateway. After the creation of the Session, the SQL Gateway server returns an identifier named SessionHandle for later interactions. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/hive-compatibility/hive-dialect/queries/sort-cluster-distribute-by/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/hive-compatibility/hive-dialect/queries/sort-cluster-distribute-by/ Sort/Cluster/Distributed by Clause # Sort By # Description # Unlike ORDER BY which guarantees a total order of output, SORT BY only guarantees the result rows with each partition is in the user specified order. So when there’s more than one partition, SORT BY may return result that’s partially ordered. Syntax # query: SELECT expression [ , ... ] FROM src sortBy sortBy: SORT BY expression colOrder [ , . //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/security/security-ssl/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/security/security-ssl/ SSL Setup # This page provides instructions on how to enable TLS/SSL authentication and encryption for network communication with and between Flink processes. NOTE: TLS/SSL authentication is not enabled by default. Internal and External Connectivity # When securing network connections between machines processes through authentication and encryption, Apache Flink differentiates between internal and external connectivity. Internal Connectivity refers to all connections made between Flink processes. These connections run Flink custom protocols. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/libs/state_processor_api/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/libs/state_processor_api/ State Processor API # Apache Flink’s State Processor API provides powerful functionality for reading, writing, and modifying savepoints and checkpoints using Flink’s DataStream API and Table API under BATCH execution. Due to the interoperability of DataStream and Table API, you can even use relational Table API or SQL queries to analyze and process state data. For example, you can take a savepoint of a running stream processing application and analyze it with a DataStream batch program to verify that the application behaves correctly. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/fault-tolerance/serialization/schema_evolution/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/fault-tolerance/serialization/schema_evolution/ State Schema Evolution # Apache Flink streaming applications are typically designed to run indefinitely or for long periods of time. As with all long-running services, the applications need to be updated to adapt to changing requirements. This goes the same for data schemas that the applications work against; they evolve along with the application. This page provides an overview of how you can evolve your state type’s data schema. The current restrictions varies across different types and state structures (ValueState, ListState, etc. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/materialized-table/statements/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/materialized-table/statements/ Materialized Table Statements # Flink SQL supports the following Materialized Table statements for now: CREATE MATERIALIZED TABLE ALTER MATERIALIZED TABLE DROP MATERIALIZED TABLE CREATE MATERIALIZED TABLE # CREATE MATERIALIZED TABLE [catalog_name.][db_name.]table_name [ ([ <table_constraint> ]) ] [COMMENT table_comment] [PARTITIONED BY (partition_column_name1, partition_column_name2, ...)] [WITH (key1=val1, key2=val2, ...)] [FRESHNESS = INTERVAL '<num>' { SECOND[S] | MINUTE[S] | HOUR[S] | DAY[S] }] [REFRESH_MODE = { CONTINUOUS | FULL }] AS <select_statement> <table_constraint>: [CONSTRAINT constraint_name] PRIMARY KEY (column_name, . //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/configuration/maven/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/configuration/maven/ How to use Maven to configure your project # This guide will show you how to configure a Flink job project with Maven, an open-source build automation tool developed by the Apache Software Foundation that enables you to build, publish, and deploy projects. You can use it to manage the entire lifecycle of your software project. Requirements # Maven 3.8.6 Java 11 Importing the project into your IDE # Once the project folder and files have been created, we recommend that you import this project into your IDE for developing and testing. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/operators/windows/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/operators/windows/ Windows # Windows are at the heart of processing infinite streams. Windows split the stream into “buckets” of finite size, over which we can apply computations. This document focuses on how windowing is performed in Flink and how the programmer can benefit to the maximum from its offered functionality. The general structure of a windowed Flink program is presented below. The first snippet refers to keyed streams, while the second to non-keyed ones. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/datastream/operators/windows/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/datastream/operators/windows/ //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/fault-tolerance/state_v2/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/fault-tolerance/state_v2/ Working with State V2 (New APIs) # In this section you will learn about the new APIs that Flink provides for writing stateful programs. Please take a look at Stateful Stream Processing to learn about the concepts behind stateful stream processing. The new state API is designed to be more flexible than the previous API. User can perform asynchronous state operations, thus making it more powerful and more efficient. The asynchronous state access is essential for the state backend to be able to handle large state sizes and to be able to spill to remote file systems when necessary. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/ha/zookeeper_ha/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/ha/zookeeper_ha/ ZooKeeper HA Services # Flink’s ZooKeeper HA services use ZooKeeper for high availability services. Flink leverages ZooKeeper for distributed coordination between all running JobManager instances. ZooKeeper is a separate service from Flink, which provides highly reliable distributed coordination via leader election and light-weight consistent state storage. Check out ZooKeeper’s Getting Started Guide for more information about ZooKeeper. Flink includes scripts to bootstrap a simple ZooKeeper installation. Configuration # In order to start an HA-cluster you have to configure the following configuration keys: //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/hive-compatibility/hive-dialect/alter/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/hive-compatibility/hive-dialect/alter/ ALTER Statements # With Hive dialect, the following ALTER statements are supported for now: ALTER DATABASE ALTER TABLE ALTER VIEW ALTER DATABASE # Description # ALTER DATABASE statement is used to change the properties or location of a database. Syntax # -- alter database's properties ALTER (DATABASE|SCHEMA) database_name SET DBPROPERTIES (property_name=property_value, ...); -- alter database's localtion ALTER (DATABASE|SCHEMA) database_name SET LOCATION hdfs_path; Synopsis # The uses of SCHEMA and DATABASE are interchangeable - they mean the same thing. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/filesystems/s3/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/filesystems/s3/ Amazon S3 # Amazon Simple Storage Service (Amazon S3) provides cloud object storage for a variety of use cases. You can use S3 with Flink for reading and writing data as well in conjunction with the streaming state backends. You can use S3 objects like regular files by specifying paths in the following format: s3://<your-bucket>/<endpoint> The endpoint can either be a single file or a directory, for example: // Read from S3 bucket FileSource<String> fileSource = FileSource. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream-v2/building_blocks/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream-v2/building_blocks/ Note: DataStream API V2 is a new set of APIs, to gradually replace the original DataStream API. It is currently in the experimental stage and is not fully available for production. Building Blocks # DataStream, Partitioning, ProcessFunction are the most fundamental elements of DataStream API and respectively represent: What are the types of data streams How data is partitioned How to perform operations / processing on data streams They are also the core parts of the fundamental primitives provided by DataStream API. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/event-time/built_in/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/event-time/built_in/ Builtin Watermark Generators # As described in Generating Watermarks, Flink provides abstractions that allow the programmer to assign their own timestamps and emit their own watermarks. More specifically, one can do so by implementing the WatermarkGenerator interface. In order to further ease the programming effort for such tasks, Flink comes with some pre-implemented timestamp assigners. This section provides a list of them. Apart from their out-of-the-box functionality, their implementation can serve as an example for custom implementations. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/config/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/config/ Configuration # All configuration can be set in Flink configuration file in the conf/ directory (see Flink Configuration File). The configuration is parsed and evaluated when the Flink processes are started. Changes to the configuration file require restarting the relevant processes. The out of the box configuration will use your default Java installation. You can manually set the environment variable JAVA_HOME or the configuration key env.java.home in Flink configuration file if you want to manually override the Java runtime to use. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/fault-tolerance/serialization/custom_serialization/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/fault-tolerance/serialization/custom_serialization/ Custom Serialization for Managed State # This page is targeted as a guideline for users who require the use of custom serialization for their state, covering how to provide a custom state serializer as well as guidelines and best practices for implementing serializers that allow state schema evolution. If you’re simply using Flink’s own serializers, this page is irrelevant and can be ignored. Using custom state serializers # When registering a managed operator or keyed state, a StateDescriptor is required to specify the state’s name, as well as information about the type of the state. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/datagen/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/datagen/ DataGen Connector # The DataGen connector provides a Source implementation that allows for generating input data for Flink pipelines. It is useful when developing locally or demoing without access to external systems such as Kafka. The DataGen connector is built-in, no additional dependencies are required. Usage # The DataGeneratorSource produces N data points in parallel. The source splits the sequence into as many parallel sub-sequences as there are parallel source subtasks. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/data_stream_api/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/data_stream_api/ DataStream API Integration # Both Table API and DataStream API are equally important when it comes to defining a data processing pipeline. The DataStream API offers the primitives of stream processing (namely time, state, and dataflow management) in a relatively low-level imperative programming API. The Table API abstracts away many internals and provides a structured and declarative API. Both APIs can work with bounded and unbounded streams. Bounded streams need to be managed when processing historical data. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/debugging/debugging_classloading/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/debugging/debugging_classloading/ Debugging Classloading # Overview of Classloading in Flink # When running Flink applications, the JVM will load various classes over time. These classes can be divided into three groups based on their origin: The Java Classpath: This is Java’s common classpath, and it includes the JDK libraries, and all code in Flink’s /lib folder (the classes of Apache Flink and some dependencies). They are loaded by AppClassLoader. The Flink Plugin Components: The plugins code in folders under Flink’s /plugins folder. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/materialized-table/deployment/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/materialized-table/deployment/ Introduction # Creating and operating materialized tables involves multiple components’ collaborative work. This document will systematically explain the complete deployment solution for Materialized Tables, covering architectural overview, environment preparation, deployment procedures, and operational practices. Architecture Introduction # Client: Could be any client that can interact with Flink SQL Gateway, such as SQL Client, Flink JDBC Driver and so on. Flink SQL Gateway: Supports creating, altering, and dropping Materialized table. It also serves as an embedded workflow scheduler to periodically refresh full mode Materialized Table. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/dynamic-kafka/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/dynamic-kafka/ Dynamic Kafka Source Experimental # Flink provides an Apache Kafka connector for reading data from Kafka topics from one or more Kafka clusters. The Dynamic Kafka connector discovers the clusters and topics using a Kafka metadata service and can achieve reading in a dynamic fashion, facilitating changes in topics and/or clusters, without requiring a job restart. This is especially useful when you need to read a new Kafka cluster/topic and/or stop reading an existing Kafka cluster/topic (cluster migration/failover/other infrastructure changes) and when you need direct integration with Hybrid Source. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/debugging/flame_graphs/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/debugging/flame_graphs/ Flame Graphs # Flame Graphs are a visualization that effectively surfaces answers to questions like: Which methods are currently consuming CPU resources? How does consumption by one method compare to the others? Which series of calls on the stack led to executing a particular method? Flame Graph Flame Graphs are constructed by sampling stack traces a number of times. Each method call is presented by a bar, where the length of the bar is proportional to the number of times it is present in the samples. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/filesystems/gcs/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/filesystems/gcs/ Google Cloud Storage # Google Cloud Storage (GCS) provides cloud storage for a variety of use cases. You can use it for reading and writing data, and for checkpoint storage when using FileSystemCheckpointStorage) with the streaming state backends. You can use GCS objects like regular files by specifying paths in the following format: gs://<your-bucket>/<endpoint> The endpoint can either be a single file or a directory, for example: // Read from GCS bucket FileSource<String> fileSource = FileSource. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/hive-compatibility/hive-dialect/queries/group-by/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/hive-compatibility/hive-dialect/queries/group-by/ Group By Clause # Description # The Group by clause is used to compute a single result from multiple input rows with given aggregation function. Hive dialect also supports enhanced aggregation features to do multiple aggregations based on the same record by using ROLLUP/CUBE/GROUPING SETS. Syntax # group_by_clause: group_by_clause_1 | group_by_clause_2 group_by_clause_1: GROUP BY group_expression [ , ... ] [ WITH ROLLUP | WITH CUBE ] group_by_clause_2: GROUP BY { group_expression | { ROLLUP | CUBE | GROUPING SETS } ( grouping_set [ , . //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/advanced/historyserver/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/advanced/historyserver/ History Server # Flink has a history server that can be used to query the statistics of completed jobs after the corresponding Flink cluster has been shut down. Furthermore, it exposes a REST API that accepts HTTP requests and responds with JSON data. Overview # The HistoryServer allows you to query the status and statistics of completed jobs that have been archived by a JobManager. After you have configured the HistoryServer and JobManager, you start and stop the HistoryServer via its corresponding startup script: //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql-gateway/hiveserver2/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql-gateway/hiveserver2/ HiveServer2 Endpoint # The Flink SQL Gateway supports deploying as a HiveServer2 Endpoint which is compatible with HiveServer2 wire protocol. This allows users to submit Hive-dialect SQL through the Flink SQL Gateway with existing Hive clients using Thrift or the Hive JDBC driver. These clients include Beeline, DBeaver, Apache Superset and so on. It is recommended to use the HiveServer2 Endpoint with a Hive Catalog and Hive dialect to get the same experience as HiveServer2. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/hive-compatibility/hive-dialect/insert/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/hive-compatibility/hive-dialect/insert/ INSERT Statements # INSERT TABLE # Description # The INSERT TABLE statement is used to insert rows into a table or overwrite the existing data in the table. The row to be inserted can be specified by value expressions or result from query. Syntax # -- Stardard syntax INSERT { OVERWRITE TABLE | INTO [TABLE] } tablename [PARTITION (partcol1[=val1], partcol2[=val2] ...) [IF NOT EXISTS]] { VALUES ( value [, . //nightlies.apache.org/flink/flink-docs-release-2.2/docs/learn-flink/datastream_api/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/learn-flink/datastream_api/ Intro to the DataStream API # The focus of this training is to broadly cover the DataStream API well enough that you will be able to get started writing streaming applications. What can be Streamed? # Flink’s DataStream APIs will let you stream anything they can serialize. Flink’s own serializer is used for basic types, i.e., String, Long, Integer, Boolean, Array composite types: Tuples, POJOs and Flink falls back to Kryo for other types. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/operators/joining/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/operators/joining/ Joining # Window Join # A window join joins the elements of two streams that share a common key and lie in the same window. These windows can be defined by using a window assigner and are evaluated on elements from both of the streams. The elements from both sides are then passed to a user-defined JoinFunction or FlatJoinFunction where the user can emit results that meet the join criteria. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/formats/json/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/formats/json/ JSON Format # Format: Serialization Schema Format: Deserialization Schema The JSON format allows to read and write JSON data based on an JSON schema. Currently, the JSON schema is derived from table schema. The JSON format supports append-only streams, unless you’re using a connector that explicitly support retract streams and/or upsert streams like the Upsert Kafka connector. If you need to write retract streams and/or upsert streams, we suggest you to look at CDC JSON formats like Debezium JSON and Canal JSON. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/kafka/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/kafka/ Apache Kafka Connector # Flink provides an Apache Kafka connector for reading data from and writing data to Kafka topics with exactly-once guarantees. Dependency # Apache Flink ships with a universal Kafka connector which attempts to track the latest version of the Kafka client. The version of the client it uses may change between Flink releases. Modern Kafka clients are backwards compatible with broker versions 0.10.0 or later. For details on Kafka compatibility, please refer to the official Kafka documentation. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/kafka/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/kafka/ Apache Kafka SQL Connector # Scan Source: Unbounded Sink: Streaming Append Mode The Kafka connector allows for reading data from and writing data into Kafka topics. Dependencies # There is no connector (yet) available for Flink version 2.2. The Kafka connector is not part of the binary distribution. See how to link with it for cluster execution here. How to create a Kafka table # The example below shows how to create a Kafka table: //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/security/security-kerberos/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/security/security-kerberos/ Kerberos Authentication Setup and Configuration # This document briefly describes how Flink security works in the context of various deployment mechanisms (Standalone, native Kubernetes, YARN), filesystems, connectors, and state backends. Objective # The primary goals of the Flink Kerberos security infrastructure are: to enable secure data access for jobs within a cluster via connectors (e.g. Kafka) to authenticate to ZooKeeper (if configured to use SASL) to authenticate to Hadoop components (e. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/ha/kubernetes_ha/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/ha/kubernetes_ha/ Kubernetes HA Services # Flink’s Kubernetes HA services use Kubernetes for high availability services. Kubernetes high availability services can only be used when deploying to Kubernetes. Consequently, they can be configured when using standalone Flink on Kubernetes or the native Kubernetes integration Prerequisites # In order to use Flink’s Kubernetes HA services you must fulfill the following prerequisites: Kubernetes >= 1.9. Service account with permissions to create, edit, delete ConfigMaps. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/monitoring/back_pressure/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/monitoring/back_pressure/ Monitoring Back Pressure # Flink’s web interface provides a tab to monitor the back pressure behaviour of running jobs. Back Pressure # If you see a back pressure warning (e.g. High) for a task, this means that it is producing data faster than the downstream operators can consume. Records in your job flow downstream (e.g. from sources to sinks) and back pressure is propagated in the opposite direction, up the stream. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/resource-providers/native_kubernetes/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/resource-providers/native_kubernetes/ Native Kubernetes # This page describes how to deploy Flink natively on Kubernetes. Getting Started # This Getting Started section guides you through setting up a fully functional Flink Cluster on Kubernetes. Introduction # Kubernetes is a popular container-orchestration system for automating computer application deployment, scaling, and management. Flink’s native Kubernetes integration allows you to directly deploy Flink on a running Kubernetes cluster. Moreover, Flink is able to dynamically allocate and de-allocate TaskManagers depending on the required resources because it can directly talk to Kubernetes. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/debugging/profiler/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/debugging/profiler/ Profiler # Since Flink 1.19, we support profiling the JobManager/TaskManager process interactively with async-profiler via Flink Web UI, which allows users to create a profiling instance with arbitrary intervals and event modes, e.g ITIMER, CPU, Lock, Wall-Clock and Allocation. CPU: In this mode the profiler collects stack trace samples that include Java methods, native calls, JVM code and kernel functions. ALLOCATION: In allocation profiling mode, the top frame of every call trace is the class of the allocated object, and the counter is the heap pressure (the total size of allocated TLABs or objects outside TLAB). //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/memory/mem_setup_tm/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/memory/mem_setup_tm/ Set up TaskManager Memory # The TaskManager runs user code in Flink. Configuring memory usage for your needs can greatly reduce Flink’s resource footprint and improve Job stability. The further described memory configuration is applicable starting with the release version 1.10. If you upgrade Flink from earlier versions, check the migration guide because many changes were introduced with the 1.10 release. This memory setup guide is relevant only for TaskManagers! //nightlies.apache.org/flink/flink-docs-release-2.2/docs/concepts/stateful-stream-processing/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/concepts/stateful-stream-processing/ Stateful Stream Processing # What is State? # While many operations in a dataflow simply look at one individual event at a time (for example an event parser), some operations remember information across multiple events (for example window operators). These operations are called stateful. Some examples of stateful operations: When an application searches for certain event patterns, the state will store the sequence of events encountered so far. When aggregating events per minute/hour/day, the state holds the pending aggregates. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/fault-tolerance/broadcast_state/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/fault-tolerance/broadcast_state/ The Broadcast State Pattern # In this section you will learn about how to use broadcast state in practise. Please refer to Stateful Stream Processing to learn about the concepts behind stateful stream processing. Provided APIs # To show the provided APIs, we will start with an example before presenting their full functionality. As our running example, we will use the case where we have a stream of objects of different colors and shapes and we want to find pairs of objects of the same color that follow a certain pattern, e. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/concepts/time_attributes/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/concepts/time_attributes/ Time Attributes # Flink can process data based on different notions of time. Processing time refers to the machine’s system time (also known as epoch time, e.g. Java’s System.currentTimeMillis()) that is executing the respective operation. Event time refers to the processing of streaming data based on timestamps that are attached to each row. The timestamps can encode when an event happened. For more information about time handling in Flink, see the introduction about event time and watermarks. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/concepts/time/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/concepts/time/ Timely Stream Processing # Introduction # Timely stream processing is an extension of stateful stream processing in which time plays some role in the computation. Among other things, this is the case when you do time series analysis, when doing aggregations based on certain time periods (typically called windows), or when you do event processing where the time when an event occurred is important. In the following sections we will highlight some of the topics that you should consider when working with timely Flink Applications. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/configuration/gradle/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/configuration/gradle/ How to use Gradle to configure your project # You will likely need a build tool to configure your Flink project. This guide will show you how to do so with Gradle, an open-source general-purpose build tool that can be used to automate tasks in the development process. Requirements # Gradle 7.x Java 8 (deprecated) or Java 11 Importing the project into your IDE # Once the project folder and files have been created, we recommend that you import this project into your IDE for developing and testing. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/with/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/with/ WITH clause # Batch Streaming WITH provides a way to write auxiliary statements for use in a larger query. These statements, which are often referred to as Common Table Expression (CTE), can be thought of as defining temporary views that exist just for one query. The syntax of WITH statement is: WITH <with_item_definition> [ , ... ] SELECT ... FROM ...; <with_item_defintion>: with_item_name (column_name[, ...n]) AS ( <select_query> ) The following example defines a common table expression orders_with_total and use it in a GROUP BY query. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/resource-providers/standalone/working_directory/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/resource-providers/standalone/working_directory/ Working Directory # Flink supports to configure a working directory (FLIP-198) for Flink processes (JobManager and TaskManager). The working directory is used by the processes to store information that can be recovered upon a process restart. The requirement for this to work is that the process is started with the same identity and has access to the volume on which the working directory is stored. Configuring the Working Directory # The working directories for the Flink processes are: //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/fault-tolerance/serialization/third_party_serializers/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/fault-tolerance/serialization/third_party_serializers/ 3rd Party Serializers # If you use a custom type in your Flink program which cannot be serialized by the Flink type serializer, Flink falls back to using the generic Kryo serializer. You may register your own serializer or a serialization system like Google Protobuf or Apache Thrift with Kryo. To do that, simply register the type class and the serializer via the configuration option pipeline.serialization-config: pipeline.serialization-config: - org.example.MyCustomType: {type: kryo, kryo-type: registered, class: org. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/filesystems/oss/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/filesystems/oss/ Aliyun Object Storage Service (OSS) # OSS: Object Storage Service # Aliyun Object Storage Service (Aliyun OSS) is widely used, particularly popular among China’s cloud users, and it provides cloud object storage for a variety of use cases. You can use OSS with Flink for reading and writing data as well in conjunction with the streaming state backends You can use OSS objects like regular files by specifying paths in the following format: //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/debugging/application_profiling/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/debugging/application_profiling/ Application Profiling & Debugging # Overview of Custom Logging with Apache Flink # Each standalone JobManager, TaskManager, HistoryServer, and ZooKeeper daemon redirects stdout and stderr to a file with a .out filename suffix and writes internal logging to a file with a .log suffix. Java options configured by the user in env.java.opts.all, env.java.opts.jobmanager, env.java.opts.taskmanager, env.java.opts.historyserver and env.java.opts.client can likewise define log files with use of the script variable FLINK_LOG_PREFIX and by enclosing the options in double quotes for late evaluation. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/formats/avro/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/formats/avro/ Avro format # Flink has built-in support for Apache Avro. This allows to easily read and write Avro data based on an Avro schema with Flink. The serialization framework of Flink is able to handle classes generated from Avro schemas. In order to use the Avro format the following dependencies are required for projects using a build automation tool (such as Maven or SBT). <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-avro</artifactId> <version>2.2.0</version> </dependency> In order to use the Avro format in PyFlink jobs, the following dependencies are required: PyFlink JAR Download See Python dependency management for more details on how to use JARs in PyFlink. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/formats/avro/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/formats/avro/ Avro Format # Format: Serialization Schema Format: Deserialization Schema The Apache Avro format allows to read and write Avro data based on an Avro schema. Currently, the Avro schema is derived from table schema. Dependencies # In order to use the Avro format the following dependencies are required for both projects using a build automation tool (such as Maven or SBT) and SQL Client with SQL JAR bundles. Maven dependency SQL Client &ltdependency&gt &ltgroupId&gtorg. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/filesystems/azure/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/filesystems/azure/ Azure Blob Storage # Azure Blob Storage is a Microsoft-managed service providing cloud storage for a variety of use cases. You can use Azure Blob Storage with Flink for reading and writing data as well in conjunction with the streaming state backends Flink supports accessing Azure Blob Storage using both wasb:// or abfs://. Azure recommends using abfs:// for accessing ADLS Gen2 storage accounts even though wasb:// works through backward compatibility. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/formats/azure_table_storage/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/formats/azure_table_storage/ Azure Table Storage # This example is using the HadoopInputFormat wrapper to use an existing Hadoop input format implementation for accessing Azure’s Table Storage. Download and compile the azure-tables-hadoop project. The input format developed by the project is not yet available in Maven Central, therefore, we have to build the project ourselves. Execute the following commands: git clone https://github.com/mooso/azure-tables-hadoop.git cd azure-tables-hadoop mvn clean install Setup a new Flink project using the quickstarts: curl https://flink. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/batch/batch_shuffle/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/batch/batch_shuffle/ Batch Shuffle # Overview # Flink supports a batch execution mode in both DataStream API and Table / SQL for jobs executing across bounded input. In batch execution mode, Flink offers two modes for network exchanges: Blocking Shuffle and Hybrid Shuffle. Blocking Shuffle is the default data exchange mode for batch executions. It persists all intermediate data, and can be consumed only after fully produced. Hybrid Shuffle is the next generation data exchange mode for batch executions. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/cassandra/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/cassandra/ Apache Cassandra Connector # This connector provides sinks that writes data into a Apache Cassandra database. To use this connector, add the following dependency to your project: &ltdependency&gt &ltgroupId&gtorg.apache.flink&lt/groupId&gt &ltartifactId&gtflink-connector-cassandra_2.12&lt/artifactId&gt &ltversion&gt2.2.0&lt/version&gt &lt/dependency&gt Copied to clipboard! Note that the streaming connectors are currently NOT part of the binary distribution. See how to link with them for cluster execution here. Installing Apache Cassandra # There are multiple ways to bring up a Cassandra instance on local machine: //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/fault-tolerance/checkpointing/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/fault-tolerance/checkpointing/ Checkpointing # Every function and operator in Flink can be stateful (see working with state for details). Stateful functions store data across the processing of individual elements/events, making state a critical building block for any type of more elaborate operation. In order to make state fault tolerant, Flink needs to checkpoint the state. Checkpoints allow Flink to recover state and positions in the streams to give the application the same semantics as a failure-free execution. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/formats/avro-confluent/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/formats/avro-confluent/ Confluent Avro Format # Format: Serialization Schema Format: Deserialization Schema The Avro Schema Registry (avro-confluent) format allows you to read records that were serialized by the io.confluent.kafka.serializers.KafkaAvroSerializer and to write records that can in turn be read by the io.confluent.kafka.serializers.KafkaAvroDeserializer. When reading (deserializing) a record with this format the Avro writer schema is fetched from the configured Confluent Schema Registry based on the schema version id encoded in the record while the reader schema is inferred from table schema. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream-v2/context_and_state_processing/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream-v2/context_and_state_processing/ Note: DataStream API V2 is a new set of APIs, to gradually replace the original DataStream API. It is currently in the experimental stage and is not fully available for production. Context and State Processing # Context # Unlike attributes like the name of process operation, some information(such as current key) can only be obtained when the process function is executed. In order to build a bridge between process functions and the execution engine, DataStream API provide a unified entrypoint called Runtime Context. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/create/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/create/ CREATE Statements # CREATE statements are used to register a table/view/function into current or specified Catalog. A registered table/view/function can be used in SQL queries. Flink SQL supports the following CREATE statements for now: CREATE TABLE [CREATE OR] REPLACE TABLE CREATE CATALOG CREATE DATABASE CREATE VIEW CREATE FUNCTION CREATE MODEL Run a CREATE statement # Java CREATE statements can be executed with the executeSql() method of the TableEnvironment. The executeSql() method returns ‘OK’ for a successful CREATE operation, otherwise will throw an exception. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/formats/csv/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/formats/csv/ CSV format # To use the CSV format you need to add the Flink CSV dependency to your project: <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-csv</artifactId> <version>2.2.0</version> </dependency> For PyFlink users, you could use it directly in your jobs. Flink supports reading CSV files using CsvReaderFormat. The reader utilizes Jackson library and allows passing the corresponding configuration for the CSV schema and parsing options. CsvReaderFormat can be initialized and used like this: CsvReaderFormat<SomePojo> csvFormat = CsvReaderFormat. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/learn-flink/etl/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/learn-flink/etl/ Data Pipelines & ETL # One very common use case for Apache Flink is to implement ETL (extract, transform, load) pipelines that take data from one or more sources, perform some transformations and/or enrichments, and then store the results somewhere. In this section we are going to look at how to use Flink’s DataStream API to implement this kind of application. Note that Flink’s Table and SQL APIs are well suited for many ETL use cases. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/security/security-delegation-token/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/security/security-delegation-token/ Delegation Tokens # This document aims to explain and demystify delegation tokens as they are used by Flink. Before we into the details here is the high level architecture diagram: What Are Delegation Tokens and Why Use Them? # Delegation tokens (DTs from now on) are authentication tokens used by some services to replace long-lived credentials. Many services in the Hadoop ecosystem have support for DTs, since they have some very desirable advantages over long-lived credentials: //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/resource-providers/standalone/docker/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/resource-providers/standalone/docker/ Docker Setup # Getting Started # This Getting Started section guides you through the local setup (on one machine, but in separate containers) of a Flink cluster using Docker containers. Introduction # Docker is a popular container runtime. There are official Docker images for Apache Flink available on Docker Hub. You can use the Docker images to deploy a Session or Application cluster on Docker. This page focuses on the setup of Flink on Docker and Docker Compose. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/concepts/flink-architecture/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/concepts/flink-architecture/ Flink Architecture # Flink is a distributed system and requires effective allocation and management of compute resources in order to execute streaming applications. It integrates with all common cluster resource managers such as Hadoop YARN and Kubernetes, but can also be set up to run as a standalone cluster or even as a library. This section contains an overview of Flink’s architecture and describes how its main components interact to execute applications and recover from failures. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/formats/hadoop/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/formats/hadoop/ Hadoop formats # Project Configuration # Support for Hadoop is contained in the flink-hadoop-compatibility Maven module. Add the following dependency to your pom.xml to use hadoop <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-hadoop-compatibility</artifactId> <version>2.2.0</version> </dependency> If you want to run your Flink application locally (e.g. from your IDE), you also need to add a hadoop-client dependency such as: <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> <version>2.10.2</version> <scope>provided</scope> </dependency> Using Hadoop InputFormats # To use Hadoop InputFormats with Flink the format must first be wrapped using either readHadoopFile or createHadoopInput of the HadoopInputs utility class. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/hive/hive_read_write/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/hive/hive_read_write/ Hive Read & Write # Using the HiveCatalog, Apache Flink can be used for unified BATCH and STREAM processing of Apache Hive Tables. This means Flink can be used as a more performant alternative to Hive’s batch engine, or to continuously read and write data into and out of Hive tables to power real-time data warehousing applications. Reading # Flink supports reading data from Hive in both BATCH and STREAMING modes. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/flinkdev/ide_setup/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/flinkdev/ide_setup/ Importing Flink into an IDE # The sections below describe how to import the Flink project into an IDE for the development of Flink itself. For writing Flink programs, please refer to the Java API quickstart guides. Whenever something is not working in your IDE, try with the Maven command line first (mvn clean package -DskipTests) as it might be your IDE that has a bug or is not properly set up. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/hive-compatibility/hive-dialect/queries/join/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/hive-compatibility/hive-dialect/queries/join/ Join # Description # JOIN is used to combine rows from two relations based on join condition. Syntax # Hive Dialect supports the following syntax for joining tables: join_table: table_reference [ INNER ] JOIN table_factor [ join_condition ] | table_reference { LEFT | RIGHT | FULL } [ OUTER ] JOIN table_reference join_condition | table_reference LEFT SEMI JOIN table_reference [ ON expression ] | table_reference CROSS JOIN table_reference [ join_condition ] table_reference: table_factor | join_table table_factor: tbl_name [ alias ] | table_subquery alias | ( table_references ) join_condition: { ON expression | USING ( colName [, . //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/formats/json/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/formats/json/ Json format # To use the JSON format you need to add the Flink JSON dependency to your project: <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-json</artifactId> <version>2.2.0</version> <scope>provided</scope> </dependency> For PyFlink users, you could use it directly in your jobs. Flink supports reading/writing JSON records via the JsonSerializationSchema/JsonDeserializationSchema. These utilize the Jackson library, and support any type that is supported by Jackson, including, but not limited to, POJOs and ObjectNode. The JsonDeserializationSchema can be used with any connector that supports the DeserializationSchema. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/hive-compatibility/hive-dialect/load-data/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/hive-compatibility/hive-dialect/load-data/ Load Data Statements # Description # The LOAD DATA statement is used to load the data into a Hive table from the user specified directory or file. The load operation are currently pure copy/move operations that move data files into locations corresponding to Hive tables. Syntax # LOAD DATA [LOCAL] INPATH 'filepath' [OVERWRITE] INTO TABLE tablename [PARTITION (partcol1=val1, partcol2=val2 ...)]; Parameters # filepath The filepath can be: a relative path, such as warehouse/data1 an absolute path, such as /user/hive/warehouse/data1 a full URL with schema and (optionally) an authority, such as hdfs://namenode:9000/user/hive/warehouse/data1 The filepath can refer to a file (in which case, only the single file is loaded) or it can be a directory (in which case, all the files from the directory are loaded). //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/advanced/logging/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/advanced/logging/ How to use logging # All Flink processes create a log text file that contains messages for various events happening in that process. These logs provide deep insights into the inner workings of Flink, and can be used to detect problems (in the form of WARN/ERROR messages) and can help in debugging them. The log files can be accessed via the Job-/TaskManager pages of the WebUI. The used Resource Provider (e. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/formats/parquet/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/formats/parquet/ Parquet format # Flink supports reading Parquet files, producing Flink RowData and producing Avro records. To use the format you need to add the flink-parquet dependency to your project: <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-parquet</artifactId> <version>2.2.0</version> </dependency> To read Avro records, you will need to add the parquet-avro dependency: <dependency> <groupId>org.apache.parquet</groupId> <artifactId>parquet-avro</artifactId> <version>1.12.2</version> <optional>true</optional> <exclusions> <exclusion> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> </exclusion> <exclusion> <groupId>it.unimi.dsi</groupId> <artifactId>fastutil</artifactId> </exclusion> </exclusions> </dependency> In order to use the Parquet format in PyFlink jobs, the following dependencies are required: PyFlink JAR Download See Python dependency management for more details on how to use JARs in PyFlink. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/operators/process_function/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/operators/process_function/ Process Function # The ProcessFunction # The ProcessFunction is a low-level stream processing operation, giving access to the basic building blocks of all (acyclic) streaming applications: events (stream elements) state (fault-tolerant, consistent, only on keyed stream) timers (event time and processing time, only on keyed stream) The ProcessFunction can be thought of as a FlatMapFunction with access to keyed state and timers. It handles events by being invoked for each event received in the input stream(s). //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/datastream/operators/process_function/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/datastream/operators/process_function/ Process Function # ProcessFunction # The ProcessFunction is a low-level stream processing operation, giving access to the basic building blocks of all (acyclic) streaming applications: events (stream elements) state (fault-tolerant, consistent, only on keyed stream) timers (event time and processing time, only on keyed stream) The ProcessFunction can be thought of as a FlatMapFunction with access to keyed state and timers. It handles events by being invoked for each event received in the input stream(s). //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/formats/protobuf/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/formats/protobuf/ Protobuf Format # Format: Serialization Schema Format: Deserialization Schema The Protocol Buffers Protobuf format allows you to read and write Protobuf data, based on Protobuf generated classes. Dependencies # In order to use the Protobuf format the following dependencies are required for both projects using a build automation tool (such as Maven or SBT) and SQL Client with SQL JAR bundles. Maven dependency SQL Client &ltdependency&gt &ltgroupId&gtorg.apache.flink&lt/groupId&gt &ltartifactId&gtflink-protobuf&lt/artifactId&gt &ltversion&gt2.2.0&lt/version&gt &lt/dependency&gt Copied to clipboard! //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/materialized-table/quickstart/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/materialized-table/quickstart/ Quickstart Guide # This guide will help you quickly understand and get started with materialized tables. It includes setting up the environment and creating, altering, and dropping materialized tables in CONTINUOUS and FULL mode. Environment Setup # Directory Preparation # Replace the example paths below with real paths on your machine. Create directories for Catalog Store and test-filesystem Catalog: # Directory for File Catalog Store to save catalog information mkdir -p {catalog_store_path} # Directory for test-filesystem Catalog to save table metadata and table data mkdir -p {catalog_path} # Directory for the default database of test-filesystem Catalog mkdir -p {catalog_path}/mydb Create directories for Checkpoints and Savepoints: mkdir -p {checkpoints_path} mkdir -p {savepoints_path} Resource Preparation # The method here is similar to the steps recorded in local installation. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/try-flink/table_api/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/try-flink/table_api/ Real Time Reporting with the Table API # Apache Flink offers a Table API as a unified, relational API for batch and stream processing, i.e., queries are executed with the same semantics on unbounded, real-time streams or bounded, batch data sets and produce the same results. The Table API in Flink is commonly used to ease the definition of data analytics, data pipelining, and ETL applications. What Will You Be Building? //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/batch/recovery_from_job_master_failure/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/batch/recovery_from_job_master_failure/ Batch jobs progress recovery from job master failures # Background # Previously, if the JobMaster fails and is terminated, one of the following two situations will occur: If high availability (HA) is disabled, the job will fail. If HA is enabled, a JobMaster failover will happen and the job will be restarted. Streaming jobs can resume from the latest successful checkpoints. Batch jobs, however, do not have checkpoints and have to start over from the beginning, losing all previously made progress. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/select/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/select/ SELECT & WHERE clause # Batch Streaming The general syntax of the SELECT statement is: SELECT select_list FROM table_expression [ WHERE boolean_expression ] The table_expression refers to any source of data. It could be an existing table, view, VALUES, or VALUE clause, the joined results of multiple existing tables, or a subquery. Assuming that the table is available in the catalog, the following would read all rows from Orders. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/memory/mem_setup_jobmanager/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/memory/mem_setup_jobmanager/ Set up JobManager Memory # The JobManager is the controlling element of the Flink Cluster. It consists of three distinct components: Resource Manager, Dispatcher and one JobMaster per running Flink Job. This guide walks you through high level and fine-grained memory configurations for the JobManager. The further described memory configuration is applicable starting with the release version 1.11. If you upgrade Flink from earlier versions, check the migration guide because many changes were introduced with the 1. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/formats/text_files/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/formats/text_files/ Text files format # Flink supports reading from text lines from a file using TextLineInputFormat. This format uses Java’s built-in InputStreamReader to decode the byte stream using various supported charset encodings. To use the format you need to add the Flink Connector Files dependency to your project: <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-files</artifactId> <version>2.2.0</version> </dependency> For PyFlink users, you could use it directly in your jobs. This format is compatible with the new Source that can be used in both batch and streaming modes. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/upsert-kafka/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/upsert-kafka/ Upsert Kafka SQL Connector # Scan Source: Unbounded Sink: Streaming Upsert Mode The Upsert Kafka connector allows for reading data from and writing data into Kafka topics in the upsert fashion. As a source, the upsert-kafka connector produces a changelog stream, where each data record represents an update or delete event. More precisely, the value in a data record is interpreted as an UPDATE of the last value for the same key, if any (if a corresponding key doesn’t exist yet, the update will be considered an INSERT). //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/concepts/versioned_tables/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/concepts/versioned_tables/ Versioned Tables # Flink SQL operates over dynamic tables that evolve, which may either be append-only or updating. Versioned tables represent a special type of updating table that remembers the past values for each key. Concept # Dynamic tables define relations over time. Often, particularly when working with metadata, a key’s old value does not become irrelevant when it changes. Flink SQL can define versioned tables over any dynamic table with a PRIMARY KEY constraint and time attribute. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/adaptive_batch/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/adaptive_batch/ Adaptive Batch Execution # This document describes the background, usage, and limitations of adaptive batch execution. Background # In the traditional Flink batch job execution process, the execution plan of a job is determined before submission. To optimize the execution plan, users and Flink’s static execution plan optimizer need to understand the job logic and accurately evaluate how the job will execute, including the data characteristics processed by each node and the data distribution of the connecting edges. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/operators/asyncio/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/operators/asyncio/ Asynchronous I/O for External Data Access # This page explains the use of Flink’s API for asynchronous I/O with external data stores. For users not familiar with asynchronous or event-driven programming, an article about Futures and event-driven programming may be useful preparation. Note: Details about the design and implementation of the asynchronous I/O utility can be found in the proposal and design document FLIP-12: Asynchronous I/O Design and Implementation. Details about the new retry support can be found in document FLIP-232: Add Retry Support For Async I/O In DataStream API. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/tasks-scheduling/balanced_tasks_scheduling/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/tasks-scheduling/balanced_tasks_scheduling/ Balanced Tasks Scheduling # This page describes the background and principle of balanced tasks scheduling, how to use it when running streaming jobs. Background # When the parallelism of all vertices within a Flink streaming job is inconsistent, the default strategy of Flink to deploy tasks sometimes leads some TaskManagers have more tasks while others have fewer tasks, resulting in excessive resource utilization at some TaskManagers that contain more tasks and becoming a bottleneck for the entire job processing. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/cli/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/cli/ Command-Line Interface # Flink provides a Command-Line Interface (CLI) bin/flink to run programs that are packaged as JAR files and to control their execution. The CLI is part of any Flink setup, available in local single node setups and in distributed setups. It connects to the running JobManager specified in Flink configuration file. Job Lifecycle Management # A prerequisite for the commands listed in this section to work is to have a running Flink deployment like Kubernetes, YARN or any other option available. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/configuration/connector/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/configuration/connector/ Connectors and Formats # Flink applications can read from and write to various external systems via connectors. It supports multiple formats in order to encode and decode data to match Flink’s data structures. An overview of available connectors and formats is available for both DataStream and Table API/SQL. Available artifacts # In order to use connectors and formats, you need to make sure Flink has access to the artifacts implementing them. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/formats/debezium/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/formats/debezium/ Debezium Format # Changelog-Data-Capture Format Format: Serialization Schema Format: Deserialization Schema Debezium is a CDC (Changelog Data Capture) tool that can stream changes in real-time from MySQL, PostgreSQL, Oracle, Microsoft SQL Server and many other databases into Kafka. Debezium provides a unified format schema for changelog and supports to serialize messages using JSON and Apache Avro. Flink supports to interpret Debezium JSON and Avro messages as INSERT/UPDATE/DELETE messages into Flink SQL system. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/drop/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/drop/ DROP Statements # DROP statements are used to remove a catalog with the given catalog name or to remove a registered table/view/function from the current or specified Catalog. Flink SQL supports the following DROP statements for now: DROP CATALOG DROP TABLE DROP DATABASE DROP VIEW DROP FUNCTION DROP MODEL Run a DROP statement # Java DROP statements can be executed with the executeSql() method of the TableEnvironment. The executeSql() method returns ‘OK’ for a successful DROP operation, otherwise will throw an exception. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/dynamodb/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/dynamodb/ Amazon DynamoDB Connector # The DynamoDB connector allows users to read/write from Amazon DynamoDB. As a source, the connector allows users to read change data capture stream from DynamoDB tables using Amazon DynamoDB Streams. As a sink, the connector allows users to write directly to Amazon DynamoDB tables using the BatchWriteItem API. Dependency # Apache Flink ships the connector for users to utilize. To use the connector, add the following Maven dependency to your project: //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/dynamodb/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/dynamodb/ Amazon DynamoDB SQL Connector # Sink: Batch Sink: Streaming Append & Upsert Mode The DynamoDB connector allows for writing data into Amazon DynamoDB. Dependencies # There is no connector (yet) available for Flink version 2.2. How to create a DynamoDB table # Follow the instructions from the Amazon DynamoDB Developer Guide to set up a DynamoDB table. The following example shows how to create a table backed by a DynamoDB table with minimum required options: //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/elastic_scaling/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/elastic_scaling/ Elastic Scaling # Historically, the parallelism of a job has been static throughout its lifecycle and defined once during its submission. Batch jobs couldn’t be rescaled at all, while Streaming jobs could have been stopped with a savepoint and restarted with a different parallelism. This page describes a new class of schedulers that allow Flink to adjust job’s parallelism at runtime, which pushes Flink one step closer to a truly cloud-native stream processor. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/elasticsearch/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/elasticsearch/ Elasticsearch Connector # This connector provides sinks that can request document actions to an Elasticsearch Index. To use this connector, add one of the following dependencies to your project, depending on the version of the Elasticsearch installation: Elasticsearch version Maven Dependency 6.x There is no connector (yet) available for Flink version 2.2. 7.x There is no connector (yet) available for Flink version 2.2. In order to use the in PyFlink jobs, the following dependencies are required: Version PyFlink JAR flink-connector-elasticsearch6 There is no SQL jar (yet) available for Flink version 2. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/finegrained_resource/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/finegrained_resource/ Fine-Grained Resource Management # Apache Flink works hard to auto-derive sensible default resource requirements for all applications out of the box. For users who wish to fine-tune their resource consumption, based on knowledge of their specific scenarios, Flink offers fine-grained resource management. This page describes the fine-grained resource management’s usage, applicable scenarios, and how it works. Note: This feature is currently an MVP (“minimum viable product”) feature and only available to DataStream API. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/firehose/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/firehose/ Amazon Kinesis Data Firehose Sink # The Firehose sink writes to Amazon Kinesis Data Firehose. Follow the instructions from the Amazon Kinesis Data Firehose Developer Guide to setup a Kinesis Data Firehose delivery stream. To use the connector, add the following Maven dependency to your project: There is no connector (yet) available for Flink version 2.2. In order to use the in PyFlink jobs, the following dependencies are required: Version PyFlink JAR flink-connector-aws-kinesis-firehose There is no SQL jar (yet) available for Flink version 2. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/firehose/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/firehose/ Amazon Kinesis Data Firehose SQL Connector # Sink: Batch Sink: Streaming Append Mode The Kinesis Data Firehose connector allows for writing data into Amazon Kinesis Data Firehose (KDF). Dependencies # There is no connector (yet) available for Flink version 2.2. How to create a Kinesis Data Firehose table # Follow the instructions from the Amazon Kinesis Data Firehose Developer Guide to set up a Kinesis Data Firehose delivery stream. The following example shows how to create a table backed by a Kinesis Data Firehose delivery stream with minimum required options: //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/operators/full_window_partition/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/operators/full_window_partition/ Full Window Partition Processing on DataStream # This page explains the use of full window partition processing API on DataStream. Flink enables both keyed and non-keyed DataStream to directly transform into PartitionWindowedStream now. The PartitionWindowedStream represents collecting all records of each subtask separately into a full window. The PartitionWindowedStream support four APIs: mapPartition, sortPartition, aggregate and reduce. Note: Details about the design and implementation of the full window partition processing can be found in the proposal and design document FLIP-380: Support Full Partition Processing On Non-keyed DataStream. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/table/udfs/python_udfs/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/table/udfs/python_udfs/ General User-defined Functions # User-defined functions are important features, because they significantly extend the expressiveness of Python Table API programs. NOTE: Python UDF execution requires Python version (3.9, 3.10, 3.11 or 3.12) with PyFlink installed. It’s required on both the client side and the cluster side. Scalar Functions # It supports to use Python scalar functions in Python Table API programs. In order to define a Python scalar function, one can extend the base class ScalarFunction in pyflink. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/concepts/glossary/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/concepts/glossary/ Glossary # Checkpoint Storage # The location where the State Backend will store its snapshot during a checkpoint (Java Heap of JobManager or Filesystem). Flink Application Cluster # A Flink Application Cluster is a dedicated Flink Cluster that only executes Flink Jobs from one Flink Application. The lifetime of the Flink Cluster is bound to the lifetime of the Flink Application. Flink Job Cluster # A Flink Job Cluster is a dedicated Flink Cluster that only executes a single Flink Job. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/hive/hive_functions/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/hive/hive_functions/ Hive Functions # Use Hive Built-in Functions via HiveModule # The HiveModule provides Hive built-in functions as Flink system (built-in) functions to Flink SQL and Table API users. For detailed information, please refer to HiveModule. Java String name = "myhive"; String version = "2.3.4"; tableEnv.loadModue(name, new HiveModule(version)); Scala val name = "myhive" val version = "2.3.4" tableEnv.loadModue(name, new HiveModule(version)); Python from pyflink.table.module import HiveModule name = "myhive" version = "2. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/internals/job_scheduling/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/internals/job_scheduling/ Jobs and Scheduling # This document briefly describes how Flink schedules jobs and how it represents and tracks job status on the JobManager. Scheduling # Execution resources in Flink are defined through Task Slots. Each TaskManager will have one or more task slots, each of which can run one pipeline of parallel tasks. A pipeline consists of multiple successive tasks, such as the n-th parallel instance of a MapFunction together with the n-th parallel instance of a ReduceFunction. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/kinesis/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/kinesis/ Amazon Kinesis Data Streams Connector # The Kinesis connector allows users to read from and write to Amazon Kinesis Data Streams. Dependency # To use this connector, add the below dependency to your project: There is no connector (yet) available for Flink version 2.2. For use in PyFlink jobs, use the following dependency: In order to use the in PyFlink jobs, the following dependencies are required: Version PyFlink JAR flink-connector-aws-kinesis-streams There is no SQL jar (yet) available for Flink version 2. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/kinesis/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/kinesis/ Amazon Kinesis Data Streams SQL Connector # Scan Source: Unbounded Sink: Batch Sink: Streaming Append Mode The Kinesis connector allows for reading data from and writing data into Amazon Kinesis Data Streams (KDS). Dependencies # There is no connector (yet) available for Flink version 2.2. The Kinesis connector is not part of the binary distribution. See how to link with it for cluster execution here. Versioning # There are two available Table API and SQL distributions for the Kinesis connector. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/resource-providers/standalone/kubernetes/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/resource-providers/standalone/kubernetes/ Kubernetes Setup # Getting Started # This Getting Started guide describes how to deploy a Session cluster on Kubernetes. Introduction # This page describes deploying a standalone Flink cluster on top of Kubernetes, using Flink’s standalone deployment. We generally recommend new users to deploy Flink on Kubernetes using native Kubernetes deployments. Apache Flink also provides a Kubernetes operator for managing Flink clusters on Kubernetes. It supports both standalone and native deployment mode and greatly simplifies deployment, configuration and the life cycle management of Flink resources on Kubernetes. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/memory/mem_tuning/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/memory/mem_tuning/ Memory tuning guide # In addition to the main memory setup guide, this section explains how to set up memory depending on the use case and which options are important for each case. Configure memory for standalone deployment # It is recommended to configure total Flink memory (taskmanager.memory.flink.size or jobmanager.memory.flink.size) or its components for standalone deployment where you want to declare how much memory is given to Flink itself. Additionally, you can adjust JVM metaspace if it causes problems. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/metrics/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/metrics/ Metrics # Flink exposes a metric system that allows gathering and exposing metrics to external systems. Registering metrics # You can access the metric system from any user function that extends RichFunction by calling getRuntimeContext().getMetricGroup(). This method returns a MetricGroup object on which you can create and register new metrics. Metric types # Flink supports Counters, Gauges, Histograms and Meters. Counter # A Counter is used to count something. The current value can be in- or decremented using inc()/inc(long n) or dec()/dec(long n). //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/mongodb/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/mongodb/ MongoDB Connector # Flink provides a MongoDB connector for reading and writing data from and to MongoDB collections with at-least-once guarantees. To use this connector, add one of the following dependencies to your project. There is no connector (yet) available for Flink version 2.2. MongoDB Source # The example below shows how to configure and create a source: Java import org.apache.flink.api.common.eventtime.WatermarkStrategy; import org.apache.flink.api.common.typeinfo.BasicTypeInfo; import org.apache.flink.api.common.typeinfo.TypeInformation; import org.apache.flink.connector.mongodb.source.MongoSource; import org.apache.flink.connector.mongodb.source.reader.deserializer.MongoDeserializationSchema; import org. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/mongodb/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/mongodb/ MongoDB SQL Connector # Scan Source: Bounded Lookup Source: Sync Mode Sink: Batch Sink: Streaming Append & Upsert Mode The MongoDB connector allows for reading data from and writing data into MongoDB. This document describes how to set up the MongoDB connector to run SQL queries against MongoDB. The connector can operate in upsert mode for exchanging UPDATE/DELETE messages with the external system using the primary key defined on the DDL. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/opensearch/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/opensearch/ Opensearch Connector # This connector provides sinks that can request document actions to an Opensearch Index. To use this connector, add the following dependency to your project: Opensearch version Maven Dependency 1.x There is no connector (yet) available for Flink version 2.2. 2.x There is no connector (yet) available for Flink version 2.2. By default, Apache Flink Opensearch Connector uses 1.3.x client libraries. You could switch to use 2.x (or upcoming 3. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/filesystems/plugins/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/filesystems/plugins/ Plugins # Plugins facilitate a strict separation of code through restricted classloaders. Plugins cannot access classes from other plugins or from Flink that have not been specifically whitelisted. This strict isolation allows plugins to contain conflicting versions of the same library without the need to relocate classes or to converge to common versions. Currently, file systems and metric reporters are pluggable but in the future, connectors, formats, and even user code should also be pluggable. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/prometheus/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/prometheus/ Prometheus Sink # This sink connector can be used to write data to Prometheus-compatible storage, using the Remote Write Prometheus interface. The Prometheus-compatible backend must support Remote Write 1.0 standard API, and the Remote Write endpoint must be enabled. This connector is not meant for sending internal Flink metrics to Prometheus. To publish Flink metrics, for monitoring health and operations of the Flink cluster, you should use Metric Reporters. To use the connector, add the following Maven dependency to your project: //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/select-distinct/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/select-distinct/ SELECT DISTINCT # Batch Streaming If SELECT DISTINCT is specified, all duplicate rows are removed from the result set (one row is kept from each group of duplicates). SELECT DISTINCT id FROM Orders For streaming queries, the required state for computing the query result might grow infinitely. State size depends on number of distinct rows. You can provide a query configuration with an appropriate state time-to-live (TTL) to prevent excessive state size. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/hive-compatibility/hive-dialect/queries/set-op/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/hive-compatibility/hive-dialect/queries/set-op/ Set Operations # Set Operations are used to combine multiple SELECT statements into a single result set. Hive dialect supports the following operations: UNION INTERSECT EXCEPT/MINUS UNION # Description # UNION/UNION DISTINCT/UNION ALL returns the rows that are found in either side. UNION and UNION DISTINCT only returns the distinct rows, while UNION ALL does not duplicate. Syntax # <query> { UNION [ ALL | DISTINCT ] } <query> [ . //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/hive-compatibility/hive-dialect/show/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/hive-compatibility/hive-dialect/show/ SHOW Statements # With Hive dialect, the following SHOW statements are supported for now: SHOW DATABASES SHOW TABLES SHOW VIEWS SHOW PARTITIONS SHOW FUNCTIONS SHOW DATABASES # Description # SHOW DATABASES statement is used to list all the databases defined in the metastore. Syntax # SHOW (DATABASES|SCHEMAS); The use of SCHEMA and DATABASE are interchangeable - they mean the same thing. SHOW TABLES # Description # SHOW TABLES statement lists all the base tables and views in the current database. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/speculative_execution/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/speculative_execution/ Speculative Execution # This page describes the background of speculative execution, how to use it, and how to check the effectiveness of it. Background # Speculative execution is a mechanism to mitigate job slowness which is caused by problematic nodes. A problematic node may have hardware problems, accident I/O busy, or high CPU load. These problems may make the hosted tasks run much slower than tasks on other nodes, and affect the overall execution time of a batch job. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/sqs/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/sqs/ Amazon SQS Sink # The SQS sink writes to Amazon SQS using the AWS v2 SDK for Java. Follow the instructions from the Amazon SQS Developer Guide to setup a SQS message queue. To use the connector, add the following Maven dependency to your project: There is no connector (yet) available for Flink version 2.2. Java Properties sinkProperties = new Properties(); // Required sinkProperties.put(AWSConfigConstants.AWS_REGION, "eu-west-1"); // Optional, provide via alternative routes e. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/learn-flink/streaming_analytics/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/learn-flink/streaming_analytics/ Streaming Analytics # Event Time and Watermarks # Introduction # Flink explicitly supports three different notions of time: event time: the time when an event occurred, as recorded by the device producing (or storing) the event ingestion time: a timestamp recorded by Flink at the moment it ingests the event processing time: the time when a specific operator in your pipeline is processing the event For reproducible results, e.g., when computing the maximum price a stock reached during the first hour of trading on a given day, you should use event time. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/user_defined_functions/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/user_defined_functions/ User-Defined Functions # Most operations require a user-defined function. This section lists different ways of how they can be specified. We also cover Accumulators, which can be used to gain insights into your Flink application. Implementing an interface # The most basic way is to implement one of the provided interfaces: class MyMapFunction implements MapFunction<String, Integer> { public Integer map(String value) { return Integer.parseInt(value); } } data.map(new MyMapFunction()); Anonymous classes # You can pass a function as an anonymous class: //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/resource-providers/yarn/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/resource-providers/yarn/ Apache Hadoop YARN # Getting Started # This Getting Started section guides you through setting up a fully functional Flink Cluster on YARN. Introduction # Apache Hadoop YARN is a resource provider popular with many data processing frameworks. Flink services are submitted to YARN’s ResourceManager, which spawns containers on machines managed by YARN NodeManagers. Flink deploys its JobManager and TaskManager instances into such containers. Flink can dynamically allocate and de-allocate TaskManager resources depending on the number of processing slots required by the job(s) running on the JobManager. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/alter/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/alter/ ALTER Statements # ALTER statements are used to modify the definition of a table, view or function that has already been registered in the Catalog, or the definition of a catalog itself. Flink SQL supports the following ALTER statements for now: ALTER TABLE ALTER VIEW ALTER DATABASE ALTER FUNCTION ALTER CATALOG ALTER MODEL Run an ALTER statement # Java ALTER statements can be executed with the executeSql() method of the TableEnvironment. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/formats/canal/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/formats/canal/ Canal Format # Changelog-Data-Capture Format Format: Serialization Schema Format: Deserialization Schema Canal is a CDC (Changelog Data Capture) tool that can stream changes in real-time from MySQL into other systems. Canal provides a unified format schema for changelog and supports to serialize messages using JSON and protobuf (protobuf is the default format for Canal). Flink supports to interpret Canal JSON messages as INSERT/UPDATE/DELETE messages into Flink SQL system. This is useful in many cases to leverage this feature, such as //nightlies.apache.org/flink/flink-docs-release-2.2/docs/learn-flink/event_driven/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/learn-flink/event_driven/ Event-driven Applications # Process Functions # Introduction # A ProcessFunction combines event processing with timers and state, making it a powerful building block for stream processing applications. This is the basis for creating event-driven applications with Flink. It is very similar to a RichFlatMapFunction, but with the addition of timers. Example # If you’ve done the hands-on exercise in the Streaming Analytics training, you will recall that it uses a TumblingEventTimeWindow to compute the sum of the tips for each driver during each hour, like this: //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/filesystem/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/filesystem/ FileSystem # This connector provides a unified Source and Sink for BATCH and STREAMING that reads or writes (partitioned) files to file systems supported by the Flink FileSystem abstraction. This filesystem connector provides the same guarantees for both BATCH and STREAMING and is designed to provide exactly-once semantics for STREAMING execution. The connector supports reading and writing a set of files from any (distributed) file system (e.g. POSIX, S3, HDFS) with a format (e. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/try-flink/flink-operations-playground/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/try-flink/flink-operations-playground/ Flink Operations Playground # There are many ways to deploy and operate Apache Flink in various environments. Regardless of this variety, the fundamental building blocks of a Flink Cluster remain the same, and similar operational principles apply. In this playground, you will learn how to manage and run Flink Jobs. You will see how to deploy and monitor an application, experience how Flink recovers from Job failure, and perform everyday operational tasks like upgrades and rescaling. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/jdbc/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/jdbc/ JDBC SQL Connector # Scan Source: Bounded Lookup Source: Sync Mode Sink: Batch Sink: Streaming Append & Upsert Mode The JDBC connector allows for reading data from and writing data into any relational databases with a JDBC driver. This document describes how to setup the JDBC connector to run SQL queries against relational databases. The JDBC sink operate in upsert mode for exchange UPDATE/DELETE messages with the external system if a primary key is defined on the DDL, otherwise, it operates in append mode and doesn’t support to consume UPDATE/DELETE messages. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/hive-compatibility/hive-dialect/queries/lateral-view/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/hive-compatibility/hive-dialect/queries/lateral-view/ Lateral View Clause # Description # Lateral view clause is used in conjunction with user-defined table generating functions(UDTF) such as explode(). A UDTF generates zero or more output rows for each input row. A lateral view first applies the UDTF to each row of base and then joins results output rows to the input rows to form a virtual table having the supplied table alias. Syntax # lateralView: LATERAL VIEW [ OUTER ] udtf( expression ) tableAlias AS columnAlias [, . //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/fault-tolerance/state_backends/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/fault-tolerance/state_backends/ State Backends # Flink provides different state backends that specify how and where state is stored. State can be located on Java’s heap or off-heap. Depending on your state backend, Flink can also manage the state for the application, meaning Flink deals with the memory management (possibly spilling to disk if necessary) to allow applications to hold very large state. By default, the Flink configuration file determines the state backend for all Flink jobs. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/internals/task_lifecycle/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/internals/task_lifecycle/ Task Lifecycle # A task in Flink is the basic unit of execution. It is the place where each parallel instance of an operator is executed. As an example, an operator with a parallelism of 5 will have each of its instances executed by a separate task. The StreamTask is the base for all different task sub-types in Flink’s streaming engine. This document goes through the different phases in the lifecycle of the StreamTask and describes the main methods representing each of these phases. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/configuration/testing/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/configuration/testing/ Dependencies for Testing # Flink provides utilities for testing your job that you can add as dependencies. DataStream API Testing # You need to add the following dependencies if you want to develop tests for a job built with the DataStream API: Maven Open the pom.xml file in your project directory and add the following in the dependencies block. &ltdependency&gt &ltgroupId&gtorg.apache.flink&lt/groupId&gt &ltartifactId&gtflink-test-utils&lt/artifactId&gt &ltversion&gt2.2.0&lt/version&gt &ltscope&gttest&lt/scope&gt &lt/dependency&gt Copied to clipboard! Check out Project configuration for more details. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/traces/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/traces/ Traces # Flink exposes a tracing system that allows gathering and exposing traces to external systems. Reporting traces # You can access the tracing system from any user function that extends RichFunction by calling getRuntimeContext().getMetricGroup(). This method returns a MetricGroup object via which you can report a new single trace with tree of spans. Reporting single Span # A Span represents some process that happened in Flink at certain point of time for a certain duration, that will be reported to a TraceReporter. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/memory/mem_trouble/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/memory/mem_trouble/ Troubleshooting # IllegalConfigurationException # If you see an IllegalConfigurationException thrown from TaskExecutorProcessUtils or JobManagerProcessUtils, it usually indicates that there is either an invalid configuration value (e.g. negative memory size, fraction that is greater than 1, etc.) or configuration conflicts. Check the documentation chapters or configuration options related to the memory components mentioned in the exception message. OutOfMemoryError: Java heap space # The exception usually indicates that the JVM Heap is too small. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/window-tvf/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/window-tvf/ Windowing table-valued functions (Windowing TVFs) # Batch Streaming Windows are at the heart of processing infinite streams. Windows split the stream into “buckets” of finite size, over which we can apply computations. This document focuses on how windowing is performed in Flink SQL and how the programmer can benefit to the maximum from its offered functionality. Apache Flink provides several window table-valued functions (TVF) to divide the elements of your table into windows, including: //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/hive-compatibility/hive-dialect/add/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/hive-compatibility/hive-dialect/add/ ADD Statements # With Hive dialect, the following ADD statements are supported for now: ADD JAR ADD JAR # Description # ADD JAR statement is used to add user jars into the classpath. Add multiple jars file in single ADD JAR statement is not supported. Syntax # ADD JAR <jar_path>; Parameters # jar_path The path of the JAR file to be added. It could be either on a local file or distributed file system. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/elasticsearch/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/elasticsearch/ Elasticsearch SQL Connector # Sink: Batch Sink: Streaming Append & Upsert Mode The Elasticsearch connector allows for writing into an index of the Elasticsearch engine. This document describes how to setup the Elasticsearch Connector to run SQL queries against Elasticsearch. The connector can operate in upsert mode for exchanging UPDATE/DELETE messages with the external system using the primary key defined on the DDL. If no primary key is defined on the DDL, the connector can only operate in append mode for exchanging INSERT only messages with external system. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/events/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/events/ Events # Flink exposes a event reporting system that allows gathering and exposing events to external systems. Reporting events # You can access the event system from any user function that extends RichFunction by calling getRuntimeContext().getMetricGroup(). This method returns a MetricGroup object via which you can report a new single event. Reporting single Event # An Event represents something that happened in Flink at certain point of time, that will be reported to a TraceReporter. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/learn-flink/fault_tolerance/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/learn-flink/fault_tolerance/ Fault Tolerance via State Snapshots # State Backends # The keyed state managed by Flink is a sort of sharded, key/value store, and the working copy of each item of keyed state is kept somewhere local to the taskmanager responsible for that key. Operator state is also local to the machine(s) that need(s) it. This state that Flink manages is stored in a state backend. Two implementations of state backends are available – one based on RocksDB, an embedded key/value store that keeps its working state on disk, and another heap-based state backend that keeps its working state in memory, on the Java heap. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/insert/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/insert/ INSERT Statement # INSERT statements are used to add rows to a table. Run an INSERT statement # Java Single INSERT statement can be executed through the executeSql() method of the TableEnvironment. The executeSql() method for INSERT statement will submit a Flink job immediately, and return a TableResult instance which associates the submitted job. Multiple INSERT statements can be executed through the addInsertSql() method of the StatementSet which can be created by the TableEnvironment. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/formats/maxwell/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/formats/maxwell/ Maxwell Format # Changelog-Data-Capture Format Format: Serialization Schema Format: Deserialization Schema Maxwell is a CDC (Changelog Data Capture) tool that can stream changes in real-time from MySQL into Kafka, Kinesis and other streaming connectors. Maxwell provides a unified format schema for changelog and supports to serialize messages using JSON. Flink supports to interpret Maxwell JSON messages as INSERT/UPDATE/DELETE messages into Flink SQL system. This is useful in many cases to leverage this feature, such as //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/metric_reporters/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/metric_reporters/ Metric Reporters # Flink allows reporting metrics to external systems. For more information about Flink’s metric system go to the metric system documentation. Metrics can be exposed to an external system by configuring one or several reporters in Flink configuration file. These reporters will be instantiated on each job and task manager when they are started. Below is a list of parameters that are generally applicable to all reporters. All properties are configured by setting metrics. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/memory/mem_migration/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/memory/mem_migration/ Migration Guide # The memory setup has changed a lot with the 1.10 release for TaskManagers and with the 1.11 release for JobManagers. Many configuration options were removed or their semantics changed. This guide will help you to migrate the TaskManager memory configuration from Flink <= 1.9 to >= 1.10 and the JobManager memory configuration from Flink <= 1.10 to >= 1.11. It is important to review this guide because the legacy and new memory configuration can result in different sizes of memory components. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/model-inference/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/model-inference/ Model Inference # Streaming Flink SQL provides the ML_PREDICT table-valued function (TVF) to perform model inference in SQL queries. This function allows you to apply machine learning models to your data streams directly in SQL. See Model Creation about how to create a model. ML_PREDICT Function # The ML_PREDICT function takes a table input, applies a model to it, and returns a new table with the model’s predictions. The function offers support for synchronous/asynchronous inference modes when the underlying model permits both. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/opensearch/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/opensearch/ Opensearch SQL Connector # Sink: Batch Sink: Streaming Append & Upsert Mode The Opensearch connector allows for writing into an index of the Opensearch engine. This document describes how to setup the Opensearch Connector to run SQL queries against Opensearch. The connector can operate in upsert mode for exchanging UPDATE/DELETE messages with the external system using the primary key defined on the DDL. If no primary key is defined on the DDL, the connector can only operate in append mode for exchanging INSERT only messages with external system. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/repls/python_shell/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/repls/python_shell/ Python REPL # Flink comes with an integrated interactive Python Shell. It can be used in a local setup as well as in a cluster setup. See the standalone resource provider page for more information about how to setup a local Flink. You can also build a local setup from source. Note The Python Shell will run the command “python”. Please refer to the Python Table API installation guide on how to set up the Python execution environments. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/rabbitmq/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/rabbitmq/ RabbitMQ Connector # License of the RabbitMQ Connector # Flink’s RabbitMQ connector defines a Maven dependency on the “RabbitMQ AMQP Java Client”, is triple-licensed under the Mozilla Public License 1.1 (“MPL”), the GNU General Public License version 2 (“GPL”) and the Apache License version 2 (“ASL”). Flink itself neither reuses source code from the “RabbitMQ AMQP Java Client” nor packages binaries from the “RabbitMQ AMQP Java Client”. Users that create and publish derivative work based on Flink’s RabbitMQ connector (thereby re-distributing the “RabbitMQ AMQP Java Client”) must be aware that this may be subject to conditions declared in the Mozilla Public License 1. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/trace_reporters/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/trace_reporters/ Trace Reporters # Flink allows reporting traces to external systems. For more information about Flink’s tracing system go to the tracing system documentation. Traces can be exposed to an external system by configuring one or several reporters in Flink configuration file. These reporters will be instantiated on each job and task manager when they are started. Below is a list of parameters that are generally applicable to all reporters. All properties are configured by setting traces. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/vector-search/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/vector-search/ Vector Search # Batch Streaming Flink SQL provides the VECTOR_SEARCH table-valued function (TVF) to perform a vector search in SQL queries. This function allows you to search similar rows according to the high-dimension vectors. VECTOR_SEARCH Function # The VECTOR_SEARCH uses a processing-time attribute to correlate rows to the latest version of data in an external table. It’s very similar to a lookup join in Flink SQL, however, the difference is VECTOR_SEARCH uses the input data vector to compare the similarity with data in the external table and return the top-k most similar rows. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/window-agg/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/window-agg/ Window Aggregation # Window TVF Aggregation # Batch Streaming Window aggregations are defined in the GROUP BY clause contains “window_start” and “window_end” columns of the relation applied Windowing TVF. Just like queries with regular GROUP BY clauses, queries with a group by window aggregation will compute a single result row per group. SELECT ... FROM <windowed_table> -- relation applied windowing TVF GROUP BY window_start, window_end, ... Unlike other aggregations on continuous tables, window aggregation do not emit intermediate results but only a final result, the total aggregation at the end of the window. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/hive-compatibility/hive-dialect/queries/window-functions/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/hive-compatibility/hive-dialect/queries/window-functions/ Window Functions # Description # Window functions are a kind of aggregation for a group of rows, referred as a window. It will return the aggregation value for each row based on the group of rows. Syntax # window_function OVER ( [ { PARTITION | DISTRIBUTE } BY colName ( [, ... ] ) ] { ORDER | SORT } BY expression [ ASC | DESC ] [ NULLS { FIRST | LAST } ] [ , . //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/analyze/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/analyze/ ANALYZE Statements # ANALYZE statements are used to collect statistics for existing tables and store the result to catalog. Only ANALYZE TABLE statements are supported now, and need to be triggered manually instead of automatically. Attention Currently, ANALYZE TABLE only supports in batch mode. Only existing table is supported, and an exception will be thrown if the table is a view or table not exists. Run an ANALYZE TABLE statement # Java ANALYZE TABLE statements can be executed with the executeSql() method of the TableEnvironment. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/state/checkpoints/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/state/checkpoints/ Checkpoints # Overview # Checkpoints make state in Flink fault tolerant by allowing state and the corresponding stream positions to be recovered, thereby giving the application the same semantics as a failure-free execution. See Checkpointing for how to enable and configure checkpoints for your program. To understand the differences between checkpoints and savepoints see checkpoints vs. savepoints. Checkpoint Storage # When checkpointing is enabled, managed state is persisted to ensure consistent recovery in case of failures. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/describe/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/describe/ DESCRIBE Statements # DESCRIBE statements are used to describe the schema of a table or a view, or the metadata of a catalog or a function, or the specified job in the Flink cluster. Run a DESCRIBE statement # Java DESCRIBE statements can be executed with the executeSql() method of the TableEnvironment. The executeSql() method returns objects for a successful DESCRIBE operation, otherwise will throw an exception. The following examples show how to run a DESCRIBE statement in TableEnvironment. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/event_reporters/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/event_reporters/ Event Reporters # Flink allows reporting events (structured logging) to external systems. For more information about Flink’s event reporting system go to the events system documentation. Events can be exposed to an external system by configuring one or several reporters in Flink configuration file. These reporters will be instantiated on each job and task manager when they are started. Below is a list of parameters that are generally applicable to all reporters. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/filesystem/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/filesystem/ FileSystem SQL Connector # This connector provides access to partitioned files in filesystems supported by the Flink FileSystem abstraction. The file system connector itself is included in Flink and does not require an additional dependency. The corresponding jar can be found in the Flink distribution inside the /lib directory. A corresponding format needs to be specified for reading and writing rows from and to a file system. The file system connector allows for reading and writing from a local or distributed filesystem. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/pubsub/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/pubsub/ Google Cloud PubSub # This connector provides a Source and Sink that can read from and write to Google Cloud PubSub. To use this connector, add the following dependency to your project: There is no connector (yet) available for Flink version 2.2. Note: This connector has been added to Flink recently. It has not received widespread testing yet. Note that the streaming connectors are currently not part of the binary distribution. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/group-agg/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/group-agg/ Group Aggregation # Batch Streaming Like most data systems, Apache Flink supports aggregate functions; both built-in and user-defined. User-defined functions must be registered in a catalog before use. An aggregate function computes a single result from multiple input rows. For example, there are aggregates to compute the COUNT, SUM, AVG (average), MAX (maximum) and MIN (minimum) over a set of rows. SELECT COUNT(*) FROM Orders For streaming queries, it is important to understand that Flink runs continuous queries that never terminate. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/hybridsource/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/hybridsource/ Hybrid Source # HybridSource is a source that contains a list of concrete sources. It solves the problem of sequentially reading input from heterogeneous sources to produce a single input stream. For example, a bootstrap use case may need to read several days worth of bounded input from S3 before continuing with the latest unbounded input from Kafka. HybridSource switches from FileSource to KafkaSource when the bounded file input finishes without interrupting the application. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/formats/ogg/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/formats/ogg/ Ogg Format # Changelog-Data-Capture Format Format: Serialization Schema Format: Deserialization Schema Oracle GoldenGate (a.k.a ogg) is a managed service providing a real-time data mesh platform, which uses replication to keep data highly available, and enabling real-time analysis. Customers can design, execute, and monitor their data replication and stream data processing solutions without the need to allocate or manage compute environments. Ogg provides a format schema for changelog and supports to serialize messages using JSON. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream-v2/time-processing/processing_timer_service/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream-v2/time-processing/processing_timer_service/ Note: DataStream API V2 is a new set of APIs, to gradually replace the original DataStream API. It is currently in the experimental stage and is not fully available for production. Processing Timer Service # The process timer service is a fundamental primitive of the Flink DataStream API, provided by Flink. It allows users to register timers for performing calculations at specific processing time points. For a comprehensive explanation of processing time, please refer to the section on Notions of Time: Event Time and Processing Time. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/rest_api/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/rest_api/ REST API # Flink has a monitoring API that can be used to query status and statistics of running jobs, as well as recent completed jobs. This monitoring API is used by Flink’s own dashboard, but is designed to be used also by custom monitoring tools. The monitoring API is a REST-ful API that accepts HTTP requests and responds with JSON data. Overview # The monitoring API is backed by a web server that runs as part of the JobManager. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/hive-compatibility/hive-dialect/set/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/hive-compatibility/hive-dialect/set/ SET Statements # Description # The SET statement sets a property which provide a ways to set variables for a session and configuration property including system variable and Hive configuration. But environment variable can’t be set via SET statement. The behavior of SET with Hive dialect is compatible to Hive’s. EXAMPLES # -- set Flink's configuration SET table.sql-dialect=default; -- set Hive's configuration SET hiveconf:k1=v1; -- set system property SET system:k2=v2; -- set vairable for current session SET hivevar:k3=v3; -- get value for configuration SET table. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/hive-compatibility/hive-dialect/queries/sub-queries/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/hive-compatibility/hive-dialect/queries/sub-queries/ Sub-Queries # Sub-Queries in the FROM Clause # Description # Hive dialect supports sub-queries in the FROM clause. The sub-query has to be given a name because every table in a FROM clause must have a name. Columns in the sub-query select list must have unique names. The columns in the sub-query select list are available in the outer query just like columns of a table. The sub-query can also be a query expression with UNION. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/truncate/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/truncate/ TRUNCATE Statements # Batch TRUNCATE statements are used to delete all rows from a table without dropping the table itself. Attention Currently, TRUNCATE statement is supported in batch mode, and it requires the target table connector implements the SupportsTruncate interface to support the row-level delete. An exception will be thrown if trying to TRUNCATE a table which has not implemented the related interface. Run a TRUNCATE statement # Java TRUNCATE statement can be executed with the executeSql() method of the TableEnvironment. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream-v2/builtin-funcs/windows/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream-v2/builtin-funcs/windows/ Note: DataStream API V2 is a new set of APIs, to gradually replace the original DataStream API. It is currently in the experimental stage and is not fully available for production. Windows # Windows are at the heart of processing infinite streams. Windows split the stream into “buckets” of finite size, over which we can apply computations. This document focuses on how windowing is performed in Flink DataStream and how the programmer can benefit to the maximum from its offered functionality. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/state/checkpointing_under_backpressure/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/state/checkpointing_under_backpressure/ Checkpointing under backpressure # Normally aligned checkpointing time is dominated by the synchronous and asynchronous parts of the checkpointing process. However, when a Flink job is running under heavy backpressure, the dominant factor in the end-to-end time of a checkpoint can be the time to propagate checkpoint barriers to all operators/subtasks. This is explained in the overview of the checkpointing process). and can be observed by high alignment time and start delay metrics. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/hive-compatibility/hive-dialect/queries/cte/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/hive-compatibility/hive-dialect/queries/cte/ Common Table Expression (CTE) # Description # A Common Table Expression (CTE) is a temporary result set derived from a query specified in a WITH clause, which immediately precedes a SELECT or INSERT keyword. The CTE is defined only with the execution scope of a single statement, and can be referred in the scope. Syntax # withClause: WITH cteClause [ , ... ] cteClause: cte_name AS (select statement) Note: //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream-v2/time-processing/event_timer_service/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream-v2/time-processing/event_timer_service/ Note: DataStream API V2 is a new set of APIs, to gradually replace the original DataStream API. It is currently in the experimental stage and is not fully available for production. Event Timer Service # The event timer service is a high-level extension of the Flink DataStream API, provided by Flink. It enables users to register timers for executing calculations at specific event time points and helps determine when to trigger windows within the Flink framework. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/explain/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/explain/ EXPLAIN Statements # EXPLAIN statements are used to explain the logical and optimized query plans of a query or an INSERT statement. Run an EXPLAIN statement # Java EXPLAIN statements can be executed with the executeSql() method of the TableEnvironment. The executeSql() method returns explain result for a successful EXPLAIN operation, otherwise will throw an exception. The following examples show how to run an EXPLAIN statement in TableEnvironment. Scala EXPLAIN statements can be executed with the executeSql() method of the TableEnvironment. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/hbase/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/hbase/ HBase SQL Connector # Scan Source: Bounded Lookup Source: Sync Mode Sink: Batch Sink: Streaming Upsert Mode The HBase connector allows for reading from and writing to an HBase cluster. This document describes how to setup the HBase Connector to run SQL queries against HBase. HBase always works in upsert mode for exchange changelog messages with the external system using a primary key defined on the DDL. The primary key must be defined on the HBase rowkey field (rowkey field must be declared). //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream-v2/builtin-funcs/joining/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream-v2/builtin-funcs/joining/ Note: DataStream API V2 is a new set of APIs, to gradually replace the original DataStream API. It is currently in the experimental stage and is not fully available for production. Joining # Join is used to merge two data streams by matching elements from both streams based on a common key, and performing calculations on the matched elements. This section will introduce the Join operation in DataStream in detail. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/over-agg/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/over-agg/ Over Aggregation # Batch Streaming OVER aggregates compute an aggregated value for every input row over a range of ordered rows. In contrast to GROUP BY aggregates, OVER aggregates do not reduce the number of result rows to a single row for every group. Instead OVER aggregates produce an aggregated value for every input row. The following query computes for every order the sum of amounts of all orders for the same product that were received within one hour before the current order. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/formats/parquet/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/formats/parquet/ Parquet Format # Format: Serialization Schema Format: Deserialization Schema The Apache Parquet format allows to read and write Parquet data. Dependencies # In order to use the Parquet format the following dependencies are required for both projects using a build automation tool (such as Maven or SBT) and SQL Client with SQL JAR bundles. Maven dependency SQL Client &ltdependency&gt &ltgroupId&gtorg.apache.flink&lt/groupId&gt &ltartifactId&gtflink-parquet&lt/artifactId&gt &ltversion&gt2.2.0&lt/version&gt &lt/dependency&gt Copied to clipboard! Download How to create a table with Parquet format # Here is an example to create a table using Filesystem connector and Parquet format. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/pulsar/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/pulsar/ Apache Pulsar Connector # Flink provides an Apache Pulsar connector for reading and writing data from and to Pulsar topics with exactly-once guarantees. Dependency # You can use the connector with the Pulsar 2.10.0 or higher. It is recommended to always use the latest Pulsar version. The details on Pulsar compatibility can be found in PIP-72. There is no connector (yet) available for Flink version 2.2. In order to use the in PyFlink jobs, the following dependencies are required: Version PyFlink JAR flink-connector-pulsar There is no SQL jar (yet) available for Flink version 2. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/state/savepoints/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/state/savepoints/ Savepoints # What is a Savepoint? # A Savepoint is a consistent image of the execution state of a streaming job, created via Flink’s checkpointing mechanism. You can use Savepoints to stop-and-resume, fork, or update your Flink jobs. Savepoints consist of two parts: a directory with (typically large) binary files on stable storage (e.g. HDFS, S3, …) and a (relatively small) meta data file. The files on stable storage represent the net data of the job’s execution state image. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/configuration/advanced/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/configuration/advanced/ Advanced Configuration Topics # Anatomy of the Flink distribution # Flink itself consists of a set of classes and dependencies that form the core of Flink’s runtime and must be present when a Flink application is started. The classes and dependencies needed to run the system handle areas such as coordination, networking, checkpointing, failover, APIs, operators (such as windowing), resource management, etc. These core classes and dependencies are packaged in the flink-dist. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/state/checkpoints_vs_savepoints/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/state/checkpoints_vs_savepoints/ Checkpoints vs. Savepoints # Overview # Conceptually, Flink’s savepoints are different from checkpoints in a way that’s analogous to how backups are different from recovery logs in traditional database systems. The primary purpose of checkpoints is to provide a recovery mechanism in case of unexpected job failures. A checkpoint’s lifecycle is managed by Flink, i.e. a checkpoint is created, owned, and released by Flink - without user interaction. Because checkpoints are being triggered often, and are relied upon for failure recovery, the two main design goals for the checkpoint implementation are i) being as lightweight to create and ii) being as fast to restore from as possible. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/jdbc/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/jdbc/ JDBC Connector # This connector provides a sink that writes data to a JDBC database. To use it, add the following dependency to your project (along with your JDBC driver): There is no connector (yet) available for Flink version 2.2. Note that the streaming connectors are currently NOT part of the binary distribution. See how to link with them for cluster execution here. A driver dependency is also required to connect to a specified database. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/joins/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/joins/ Joins # Batch Streaming Flink SQL supports complex and flexible join operations over dynamic tables. There are several different types of joins to account for the wide variety of semantics queries may require. By default, the order of joins is not optimized. Tables are joined in the order in which they are specified in the FROM clause. You can tweak the performance of your join queries, by listing the tables with the lowest update frequency first and the tables with the highest update frequency last. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/formats/orc/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/formats/orc/ Orc Format # Format: Serialization Schema Format: Deserialization Schema The Apache Orc format allows to read and write Orc data. Dependencies # In order to use the ORC format the following dependencies are required for both projects using a build automation tool (such as Maven or SBT) and SQL Client with SQL JAR bundles. Maven dependency SQL Client &ltdependency&gt &ltgroupId&gtorg.apache.flink&lt/groupId&gt &ltartifactId&gtflink-orc&lt/artifactId&gt &ltversion&gt2.2.0&lt/version&gt &lt/dependency&gt Copied to clipboard! Download How to create a table with Orc format # Here is an example to create a table using Filesystem connector and Orc format. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/hive-compatibility/hive-dialect/queries/transform/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/hive-compatibility/hive-dialect/queries/transform/ Transform Clause # Description # The TRANSFORM clause allows user to transform inputs using user-specified command or script. Syntax # query: SELECT TRANSFORM ( expression [ , ... ] ) [ inRowFormat ] [ inRecordWriter ] USING command_or_script [ AS colName [ colType ] [ , ... ] ] [ outRowFormat ] [ outRecordReader ] rowFormat : ROW FORMAT (DELIMITED [FIELDS TERMINATED BY char] [COLLECTION ITEMS TERMINATED BY char] [MAP KEYS TERMINATED BY char] [ESCAPED BY char] [LINES SEPARATED BY char] | SERDE serde_name [WITH SERDEPROPERTIES property_name=property_value, property_name=property_value, . //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/upgrading/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/upgrading/ Upgrading Applications and Flink Versions # Flink DataStream programs are typically designed to run for long periods of time such as weeks, months, or even years. As with all long-running services, Flink streaming applications need to be maintained, which includes fixing bugs, implementing improvements, or migrating an application to a Flink cluster of a later version. This document describes how to update a Flink streaming application and how to migrate a running streaming application to a different Flink cluster. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/use/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/use/ USE Statements # USE statements are used to set the current database or catalog, or change the resolution order and enabled status of module. Run a USE statement # Java USE statements can be executed with the executeSql() method of the TableEnvironment. The executeSql() method returns ‘OK’ for a successful USE operation, otherwise will throw an exception. The following examples show how to run a USE statement in TableEnvironment. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/table/udfs/vectorized_python_udfs/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/table/udfs/vectorized_python_udfs/ Vectorized User-defined Functions # Vectorized Python user-defined functions are functions which are executed by transferring a batch of elements between JVM and Python VM in Arrow columnar format. The performance of vectorized Python user-defined functions are usually much higher than non-vectorized Python user-defined functions as the serialization/deserialization overhead and invocation overhead are much reduced. Besides, users could leverage the popular Python libraries such as Pandas, Numpy, etc for the vectorized Python user-defined functions implementation. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/window-join/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/window-join/ Window Join # Batch Streaming A window join adds the dimension of time into the join criteria themselves. In doing so, the window join joins the elements of two streams that share a common key and are in the same window. The semantic of window join is same to the DataStream window join For streaming queries, unlike other joins on continuous tables, window join does not emit intermediate results but only emits final results at the end of the window. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/sources/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/sources/ Data Sources # This page describes Flink’s Data Source API and the concepts and architecture behind it. Read this, if you are interested in how data sources in Flink work, or if you want to implement a new Data Source. If you are looking for pre-defined source connectors, please check the Connector Docs. Data Source Concepts # Core Components A Data Source has three core components: Splits, the SplitEnumerator, and the SourceReader. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/internals/filesystems/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/internals/filesystems/ File Systems # Flink has its own file system abstraction via the org.apache.flink.core.fs.FileSystem class. This abstraction provides a common set of operations and minimal guarantees across various types of file system implementations. The FileSystem’s set of available operations is quite limited, in order to support a wide range of file systems. For example, appending to or mutating existing files is not supported. File systems are identified by a file system scheme, such as file://, hdfs://, etc. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/production_ready/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/production_ready/ Production Readiness Checklist # The production readiness checklist provides an overview of configuration options that should be carefully considered before bringing an Apache Flink job into production. While the Flink community has attempted to provide sensible defaults for each configuration, it is important to review this list and ensure the options chosen are sufficient for your needs. Set An Explicit Max Parallelism # The max parallelism, set on a per-job and per-operator granularity, determines the maximum parallelism to which a stateful operator can scale. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/formats/raw/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/formats/raw/ Raw Format # Format: Serialization Schema Format: Deserialization Schema The Raw format allows to read and write raw (byte based) values as a single column. Note: this format encodes null values as null of byte[] type. This may have limitation when used in upsert-kafka, because upsert-kafka treats null values as a tombstone message (DELETE on the key). Therefore, we recommend avoiding using upsert-kafka connector and the raw format as a value. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/set-ops/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/set-ops/ Set Operations # Batch Streaming UNION # UNION and UNION ALL return the rows that are found in either table. UNION takes only distinct rows while UNION ALL does not remove duplicates from the result rows. Flink SQL> create view t1(s) as values ('c'), ('a'), ('b'), ('b'), ('c'); Flink SQL> create view t2(s) as values ('d'), ('e'), ('a'), ('b'), ('b'); Flink SQL> (SELECT s FROM t1) UNION (SELECT s FROM t2); +---+ | s| +---+ | c| | a| | b| | d| | e| +---+ Flink SQL> (SELECT s FROM t1) UNION ALL (SELECT s FROM t2); +---+ | c| +---+ | c| | a| | b| | b| | c| | d| | e| | a| | b| | b| +---+ INTERSECT # INTERSECT and INTERSECT ALL return the rows that are found in both tables. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/show/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/show/ SHOW Statements # SHOW statements are used to list objects within their corresponding parent, such as catalogs, databases, tables and views, columns, functions, and modules. See the individual commands for more details and additional options. SHOW CREATE statements are used to print a DDL statement with which a given object can be created. The currently ‘SHOW CREATE’ statement is only available in printing DDL statement of the given table and view. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/hive-compatibility/hive-dialect/queries/table-sample/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/hive-compatibility/hive-dialect/queries/table-sample/ Table Sample # Description # The TABLESAMPLE statement is used to sample rows from the table. Syntax # TABLESAMPLE ( num_rows ROWS ) Note: Currently, only sample specific number of rows is supported. Parameters # num_rows ROWS num_rows is a constant positive to specify how many rows to sample. Examples # SELECT * FROM src TABLESAMPLE (5 ROWS) //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream-v2/watermark/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream-v2/watermark/ Note: DataStream API V2 is a new set of APIs, to gradually replace the original DataStream API. It is currently in the experimental stage and is not fully available for production. Watermark # Before introducing Watermark, users should be aware that Watermark in DataStream V2 does not refer to the original Watermark that measure progress in event time, but is a special event that can be customized by the user and can be propagated along the streams. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/sinks/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/sinks/ Data Sinks # This page describes Flink’s Data Sink API and the concepts and architecture behind it. Read this, if you are interested in how data sinks in Flink work, or if you want to implement a new Data Sink. If you are looking for pre-defined sink connectors, please check the Connector Docs. The Data Sink API # This section describes the major interfaces of the new Sink API introduced in FLIP-191 and FLIP-372, and provides tips to the developers on the Sink development. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/load/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/load/ LOAD Statements # LOAD statements are used to load a built-in or user-defined module. Run a LOAD statement # Java LOAD statements can be executed with the executeSql() method of the TableEnvironment. The executeSql() method returns ‘OK’ for a successful LOAD operation; otherwise, it will throw an exception. The following examples show how to run a LOAD statement in TableEnvironment. Scala LOAD statements can be executed with the executeSql() method of the TableEnvironment. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/orderby/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/orderby/ ORDER BY clause # Batch Streaming The ORDER BY clause causes the result rows to be sorted according to the specified expression(s). If two rows are equal according to the leftmost expression, they are compared according to the next expression and so on. If they are equal according to all specified expressions, they are returned in an implementation-dependent order. When running in streaming mode, the primary sort order of a table must be ascending on a time attribute. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/state/state_backends/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/state/state_backends/ State Backends # Programs written in the Data Stream API often hold state in various forms: Windows gather elements or aggregates until they are triggered Transformation functions may use the key/value state interface to store values Transformation functions may implement the CheckpointedFunction interface to make their local variables fault tolerant See also state section in the streaming API guide. When checkpointing is activated, such state is persisted upon checkpoints to guard against data loss and recover consistently. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/datagen/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/datagen/ DataGen SQL Connector # Scan Source: Bounded Scan Source: UnBounded The DataGen connector allows for creating tables based on in-memory data generation. This is useful when developing queries locally without access to external systems such as Kafka. Tables can include Computed Column syntax which allows for flexible record generation. The DataGen connector is built-in, no additional dependencies are required. Usage # By default, a DataGen table will create an unbounded number of rows with a random value for each column. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/limit/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/limit/ LIMIT clause # Batch LIMIT clause constrains the number of rows returned by the SELECT statement. In general, this clause is used in conjunction with ORDER BY to ensure that the results are deterministic. The following example selects the first 3 rows in Orders table. SELECT * FROM Orders ORDER BY orderTime LIMIT 3 Back to top //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/state/large_state_tuning/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/state/large_state_tuning/ Tuning Checkpoints and Large State # This page gives a guide how to configure and tune applications that use large state. Overview # For Flink applications to run reliably at large scale, two conditions must be fulfilled: The application needs to be able to take checkpoints reliably The resources need to be sufficient catch up with the input data streams after a failure The first sections discuss how to get well performing checkpoints at scale. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/unload/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/unload/ UNLOAD Statements # UNLOAD statements are used to unload a built-in or user-defined module. Run a UNLOAD statement # Java UNLOAD statements can be executed with the executeSql() method of the TableEnvironment. The executeSql() method returns ‘OK’ for a successful UNLOAD operation; otherwise it will throw an exception. The following examples show how to run a UNLOAD statement in TableEnvironment. Scala UNLOAD statements can be executed with the executeSql() method of the TableEnvironment. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/print/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/print/ Print SQL Connector # Sink The Print connector allows for writing every row to the standard output or standard error stream. It is designed for: Easy test for streaming job. Very useful in production debugging. Four possible format options: Print Condition1 Condition2 PRINT_IDENTIFIER:taskId> output PRINT_IDENTIFIER provided parallelism > 1 PRINT_IDENTIFIER> output PRINT_IDENTIFIER provided parallelism == 1 taskId> output no PRINT_IDENTIFIER provided parallelism > 1 output no PRINT_IDENTIFIER provided parallelism == 1 The output string format is “$row_kind(f0,f1,f2…)”, row_kind is the short string of RowKind, example is: “+I(1,1)”. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/set/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/set/ SET Statements # SET statements are used to modify the configuration or list the configuration. Run a SET statement # SQL CLI SET statements can be executed in SQL CLI. The following examples show how to run a SET statement in SQL CLI. SQL CLI Flink SQL> SET 'table.local-time-zone' = 'Europe/Berlin'; [INFO] Session property has been set. Flink SQL> SET; 'table.local-time-zone' = 'Europe/Berlin' Syntax # SET ('key' = 'value')? If no key and value are specified, it just prints all the properties. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/topn/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/topn/ Top-N # Batch Streaming Top-N queries ask for the N smallest or largest values ordered by columns. Both smallest and largest values sets are considered Top-N queries. Top-N queries are useful in cases where the need is to display only the N bottom-most or the N top- most records from batch/streaming table on a condition. This result set can be used for further analysis. Flink uses the combination of a OVER window clause and a filter condition to express a Top-N query. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/blackhole/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/blackhole/ BlackHole SQL Connector # Sink: Bounded Sink: UnBounded The BlackHole connector allows for swallowing all input records. It is designed for: high performance testing. UDF to output, not substantive sink. Just like /dev/null device on Unix-like operating systems. The BlackHole connector is built-in. How to create a BlackHole table # CREATE TABLE blackhole_table ( f0 INT, f1 INT, f2 STRING, f3 DOUBLE ) WITH ( 'connector' = 'blackhole' ); Alternatively, it may be based on an existing schema using the LIKE Clause. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/reset/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/reset/ RESET Statements # RESET statements are used to reset the configuration to the default. Run a RESET statement # SQL CLI RESET statements can be executed in SQL CLI. The following examples show how to run a RESET statement in SQL CLI. SQL CLI Flink SQL> RESET 'table.planner'; [INFO] Session property has been reset. Flink SQL> RESET; [INFO] All session properties have been set to their default values. Syntax # RESET ('key')? //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/window-topn/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/window-topn/ Window Top-N # Batch Streaming Window Top-N is a special Top-N which returns the N smallest or largest values for each window and other partitioned keys. For streaming queries, unlike regular Top-N on continuous tables, window Top-N does not emit intermediate results but only a final result, the total top N records at the end of the window. Moreover, window Top-N purges all intermediate state when no longer needed. Therefore, window Top-N queries have better performance if users don’t need results updated per record. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/deduplication/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/deduplication/ Deduplication # Batch Streaming Deduplication removes rows that duplicate over a set of columns, keeping only the first one or the last one. In some cases, the upstream ETL jobs are not end-to-end exactly-once; this may result in duplicate records in the sink in case of failover. However, the duplicate records will affect the correctness of downstream analytical jobs - e.g. SUM, COUNT - so deduplication is needed before further analysis. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/jar/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/jar/ JAR Statements # JAR statements are used to add user jars into the classpath or remove user jars from the classpath or show added jars in the classpath in the runtime. Flink SQL supports the following JAR statements for now: ADD JAR SHOW JARS REMOVE JAR Run a JAR statement # SQL CLI The following examples show how to run JAR statements in SQL CLI. SQL CLI Flink SQL> ADD JAR '/path/hello. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/job/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/job/ JOB Statements # Job statements are used for management of Flink jobs. Flink SQL supports the following JOB statements for now: SHOW JOBS DESCRIBE JOB STOP JOB Run a JOB statement # SQL CLI The following examples show how to run JOB statements in SQL CLI. SQL CLI Flink SQL> SHOW JOBS; +----------------------------------+----------+---------+-------------------------+ | job id | job name | status | start time | +----------------------------------+----------+---------+-------------------------+ | 228d70913eab60dda85c5e7f78b5782c | myjob | RUNNING | 2023-02-11T05:03:51. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/window-deduplication/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/window-deduplication/ Window Deduplication # Streaming Window Deduplication is a special Deduplication which removes rows that duplicate over a set of columns, keeping the first one or the last one for each window and partitioned keys. For streaming queries, unlike regular Deduplicate on continuous tables, Window Deduplication does not emit intermediate results but only a final result at the end of the window. Moreover, window Deduplication purges all intermediate state when no longer needed. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/match_recognize/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/match_recognize/ Pattern Recognition # Streaming It is a common use case to search for a set of event patterns, especially in case of data streams. Flink comes with a complex event processing (CEP) library which allows for pattern detection in event streams. Furthermore, Flink’s SQL API provides a relational way of expressing queries with a large set of built-in functions and rule-based optimizations that can be used out of the box. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/update/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/update/ UPDATE Statements # UPDATE statement is used to perform row-level updating on the target table according to the filter if provided. Attention Currently, UPDATE statement only supports in batch mode, and it requires the target table connector implements the SupportsRowLevelUpdate interface to support the row-level update. An exception will be thrown if trying to UPDATE the table which has not implements the related interface. Currently, there is no existing connector maintained by flink has supported UPDATE yet. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/delete/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/delete/ DELETE Statements # DELETE statement is used to perform row-level deletion on the target table according to the filter if provided. Attention Currently, DELETE statement only supports in batch mode, and it requires the target table connector implements the SupportsRowLevelDelete interface to support the row-level delete. An exception will be thrown if trying to DELETE the table which has not implements the related interface. Currently, there is no existing connector maintained by flink has supported DELETE yet. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/time-travel/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/queries/time-travel/ Time Travel # Batch Streaming The syntax of time travel is used for querying historical data. It allows users to specify a point in time and query the corresponding table data. Attention Currently, time travel requires the corresponding catalog that the table belongs to implementing the getTable(ObjectPath tablePath, long timestamp) method. See more details in Catalog. The syntax with time travel clause is: SELECT select_list FROM table_name FOR SYSTEM_TIME AS OF timestamp_expression Parameter Specification: //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/call/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sql/call/ Call Statements # Call statements are used to call a stored procedure which is usually provided to perform data manipulation or administrative tasks. Attention Currently, Call statements require the procedure called to exist in the corresponding catalog. So, please make sure the procedure exists in the catalog. If it doesn’t exist, it’ll throw an exception. You may need to refer to the doc of the catalog to see the available procedures. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/state/disaggregated_state/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/state/disaggregated_state/ Disaggregated State Management # Overview # For the first ten years of Flink, the state management is based on memory or local disk of the TaskManager. This approach works well for most use cases, but it has some limitations: Local Disk Constraints: The state size is limited by the memory or disk size of the TaskManager. Spiky Resource Usage: The local state model triggers periodic CPU and network I/O bursts during checkpointing or SST files compaction. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/flinkdev/building/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/flinkdev/building/ Building Flink from Source # This page covers how to build Flink 2.2.0 from sources. Build Flink # In order to build Flink you need the source code. Either download the source of a release or clone the git repository. In addition you need Maven 3.8.6 and a JDK (Java Development Kit). Flink requires Java 11 to build. To clone from git, enter: git clone https://github.com/apache/flink.git The simplest way of building Flink is by running: //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/types/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/types/ Data Types # Flink SQL has a rich set of native data types available to users. Data Type # A data type describes the logical type of a value in the table ecosystem. It can be used to declare input and/or output types of operations. Flink’s data types are similar to the SQL standard’s data type terminology but also contain information about the nullability of a value for efficient handling of scalar expressions. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/execution/packaging/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/execution/packaging/ Program Packaging and Distributed Execution # As described earlier, Flink programs can be executed on clusters by using a remote environment. Alternatively, programs can be packaged into JAR Files (Java Archives) for execution. Packaging the program is a prerequisite to executing them through the command line interface. Packaging Programs # To support execution from a packaged JAR file via the command line or web interface, a program must use the environment obtained by StreamExecutionEnvironment. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/table_api_tutorial/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/table_api_tutorial/ Table API Tutorial # Apache Flink offers a Table API as a unified, relational API for batch and stream processing, i.e., queries are executed with the same semantics on unbounded, real-time streams or bounded, batch data sets and produce the same results. The Table API in Flink is commonly used to ease the definition of data analytics, data pipelining, and ETL applications. What Will You Be Building? # In this tutorial, you will learn how to build a pure Python Flink Table API pipeline. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/datastream_tutorial/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/datastream_tutorial/ DataStream API Tutorial # Apache Flink offers a DataStream API for building robust, stateful streaming applications. It provides fine-grained control over state and time, which allows for the implementation of advanced event-driven systems. In this step-by-step guide, you’ll learn how to build a simple streaming application with PyFlink and the DataStream API. What Will You Be Building? # In this tutorial, you will learn how to write a simple Python DataStream pipeline. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/timezone/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/timezone/ Time Zone # Flink provides rich data types for Date and Time, including DATE, TIME, TIMESTAMP, TIMESTAMP_LTZ, INTERVAL YEAR TO MONTH, INTERVAL DAY TO SECOND (please see Date and Time for detailed information). Flink supports setting time zone in session level (please see table.local-time-zone for detailed information). These timestamp data types and time zone support of Flink make it easy to process business data across time zones. TIMESTAMP vs TIMESTAMP_LTZ # TIMESTAMP type # TIMESTAMP(p) is an abbreviation for TIMESTAMP(p) WITHOUT TIME ZONE, the precision p supports range is from 0 to 9, 6 by default. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/datastream/data_types/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/datastream/data_types/ Data Types # In Apache Flink’s Python DataStream API, a data type describes the type of a value in the DataStream ecosystem. It can be used to declare input and output types of operations and informs the system how to serialize elements. Pickle Serialization # If the type has not been declared, data would be serialized or deserialized using Pickle. For example, the program below specifies no data types. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/table/intro_to_table_api/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/table/intro_to_table_api/ Intro to the Python Table API # This document is a short introduction to the PyFlink Table API, which is used to help novice users quickly understand the basic usage of PyFlink Table API. For advanced usage, please refer to other documents in this user guide. Common Structure of Python Table API Program # All Table API and SQL programs, both batch and streaming, follow the same pattern. The following code example shows the common structure of Table API and SQL programs. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/table/table_environment/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/table/table_environment/ TableEnvironment # This document is an introduction of PyFlink TableEnvironment. It includes detailed descriptions of every public interface of the TableEnvironment class. Create a TableEnvironment # The recommended way to create a TableEnvironment is to create from an EnvironmentSettings object: from pyflink.common import Configuration from pyflink.table import EnvironmentSettings, TableEnvironment # create a streaming TableEnvironment config = Configuration() config.set_string('execution.buffer-timeout', '1 min') env_settings = EnvironmentSettings \ .new_instance() \ .in_streaming_mode() \ .with_configuration(config) \ . //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/dependency_management/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/dependency_management/ Dependency Management # There are requirements to use dependencies inside the Python API programs. For example, users may need to use third-party Python libraries in Python user-defined functions. In addition, in scenarios such as machine learning prediction, users may want to load a machine learning model inside the Python user-defined functions. When the PyFlink job is executed locally, users could install the third-party Python libraries into the local Python environment, download the machine learning model to local, etc. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/table/operations/operations/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/table/operations/operations/ //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/execution/parallel/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/execution/parallel/ Parallel Execution # This section describes how the parallel execution of programs can be configured in Flink. A Flink program consists of multiple tasks (transformations/operators, data sources, and sinks). A task is split into several parallel instances for execution and each parallel instance processes a subset of the task’s input data. The number of parallel instances of a task is called its parallelism. If you want to use savepoints you should also consider setting a maximum parallelism (or max parallelism). //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/table/operations/row_based_operations/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/table/operations/row_based_operations/ Row-based Operations # This page describes how to use row-based operations in PyFlink Table API. Map # Performs a map operation with a python general scalar function or vectorized scalar function. The output will be flattened if the output type is a composite type. from pyflink.common import Row from pyflink.table import EnvironmentSettings, TableEnvironment from pyflink.table.expressions import col from pyflink.table.udf import udf env_settings = EnvironmentSettings.in_batch_mode() table_env = TableEnvironment.create(env_settings) table = table_env. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/datastream/state/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/datastream/state/ //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/tableapi/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/tableapi/ Table API # The Table API is a unified, relational API for stream and batch processing. Table API queries can be run on batch or streaming input without modifications. The Table API is a super set of the SQL language and is specially designed for working with Apache Flink. The Table API is a language-integrated API for Scala, Java and Python. Instead of specifying queries as String values as common with SQL, Table API queries are defined in a language-embedded style in Java, Scala or Python with IDE support like autocompletion and syntax validation. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/table/python_types/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/table/python_types/ Data Types # This page describes the data types supported in PyFlink Table API. Data Type # A data type describes the logical type of a value in the table ecosystem. It can be used to declare input and/or output types of Python user-defined functions. Users of the Python Table API work with instances of pyflink.table.types.DataType within the Python Table API or when defining user-defined functions. A DataType instance declares the logical type which does not imply a concrete physical representation for transmission or storage. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/table/system_functions/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/table/system_functions/ //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/functions/systemfunctions/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/functions/systemfunctions/ System (Built-in) Functions # Flink Table API & SQL provides users with a set of built-in functions for data transformations. This page gives a brief overview of them. If a function that you need is not supported yet, you can implement a user-defined function. If you think that the function is general enough, please open a Jira issue for it with a detailed description. Scalar Functions # The scalar functions take zero, one or more values as the input and return a single value as the result. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/side_output/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/side_output/ Side Outputs # In addition to the main stream that results from DataStream operations, you can also produce any number of additional side output result streams. The type of data in the result streams does not have to match the type of data in the main stream and the types of the different side outputs can also differ. This operation can be useful when you want to split a stream of data where you would normally have to replicate the stream and then filter out from each stream the data that you don’t want to have. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/python_execution_mode/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/python_execution_mode/ Execution Mode # The Python API supports different runtime execution modes from which you can choose depending on the requirements of your use case and the characteristics of your job. The Python runtime execution mode defines how the Python user-defined functions will be executed. Prior to release-1.15, there is the only execution mode called PROCESS execution mode. The PROCESS mode means that the Python user-defined functions will be executed in separate Python processes. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/table/conversion_of_pandas/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/table/conversion_of_pandas/ Conversions between PyFlink Table and Pandas DataFrame # PyFlink Table API supports conversion between PyFlink Table and Pandas DataFrame. Convert Pandas DataFrame to PyFlink Table # Pandas DataFrames can be converted into a PyFlink Table. Internally, PyFlink will serialize the Pandas DataFrame using Arrow columnar format on the client. The serialized data will be processed and deserialized in Arrow source during execution. The Arrow source can also be used in streaming jobs, and is integrated with checkpointing to provide exactly-once guarantees. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/table/conversion_of_data_stream/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/table/conversion_of_data_stream/ //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/procedures/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/procedures/ Procedures # Flink Table API & SQL empowers users to perform data manipulation and administrative tasks with procedures. Procedures can run FLINK jobs with the provided StreamExecutionEnvironment, making them more powerful and flexible. Implementation Guide # To call a procedure, it must be available in a catalog. To provide procedures in a catalog, you need to implement the procedure and then return it using the Catalog.getProcedure(ObjectPath procedurePath) method. The following steps will guild you on how to implement and provide a procedure in a catalog. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/table/sql/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/table/sql/ //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/application_parameters/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/application_parameters/ Handling Application Parameters # Handling Application Parameters # Almost all Flink applications, both batch and streaming, rely on external configuration parameters. They are used to specify input and output sources (like paths or addresses), system parameters (parallelism, runtime configuration), and application specific parameters (typically used within user functions). Flink provides a simple utility called ParameterTool to provide some basic tooling for solving these problems. Please note that you don’t have to use the ParameterTool described here. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/state/task_failure_recovery/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/state/task_failure_recovery/ Task Failure Recovery # When a task failure happens, Flink needs to restart the failed task and other affected tasks to recover the job to a normal state. Restart strategies and failover strategies are used to control the task restarting. Restart strategies decide whether and when the failed/affected tasks can be restarted. Failover strategies decide which tasks should be restarted to recover the job. Restart Strategies # The cluster can be started with a default restart strategy which is always used when no job specific restart strategy has been defined. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/functions/udfs/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/functions/udfs/ User-defined Functions # User-defined functions (UDFs) are extension points to call frequently used logic or custom logic that cannot be expressed otherwise in queries. User-defined functions can be implemented in a JVM language (such as Java or Scala) or Python. An implementer can use arbitrary third party libraries within a UDF. This page will focus on JVM-based languages, please refer to the PyFlink documentation for details on writing general and vectorized UDFs in Python. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/functions/ptfs/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/functions/ptfs/ Process Table Functions (PTFs) # Process Table Functions (PTFs) are the most powerful function kind for Flink SQL and Table API. They enable implementing user-defined operators that can be as feature-rich as built-in operations. PTFs can take (partitioned) tables to produce a new table. They have access to Flink’s managed state, event-time and timer services, and underlying table changelogs. Conceptually, a PTF is itself a user-defined function that is a superset of all other user-defined functions. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/table/catalogs/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/table/catalogs/ //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/modules/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/modules/ Modules # Modules allow users to extend Flink’s built-in objects, such as defining functions that behave like Flink built-in functions. They are pluggable, and while Flink provides a few pre-built modules, users can write their own. For example, users can define their own geo functions and plug them into Flink as built-in functions to be used in Flink SQL and Table APIs. Another example is users can load an out-of-shelf Hive module to use Hive built-in functions as Flink built-in functions. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/catalogs/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/catalogs/ Catalogs # Catalogs provide metadata, such as databases, tables, partitions, views, and functions and information needed to access data stored in a database or other external systems. One of the most crucial aspects of data processing is managing metadata. It may be transient metadata like temporary tables, or UDFs registered against the table environment. Or permanent metadata, like that in a Hive Metastore. Catalogs provide a unified API for managing metadata and making it accessible from the Table API and SQL Queries. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/jdbcdriver/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/jdbcdriver/ Flink JDBC Driver # The Flink JDBC Driver is a Java library for enabling clients to send Flink SQL to your Flink cluster via the SQL Gateway. You can also use the Hive JDBC Driver with Flink. This is beneficial if you are running Hive dialect SQL and want to make use of the Hive Catalog. To use Hive JDBC with Flink you need to run the SQL Gateway with the HiveServer2 endpoint. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/olap_quickstart/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/olap_quickstart/ OLAP Quickstart # OLAP (OnLine Analysis Processing) is a key technology in the field of data analysis, it is generally used to perform complex queries on large data sets with latencies in seconds. Now Flink can not only support streaming and batch computing, but also supports users to deploy it as an OLAP computing service. This page will show you how to quickly set up a local Flink OLAP service, and will also introduce some best practices helping you deploy Flink OLAP service in production. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sqlclient/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sqlclient/ SQL Client # Flink’s Table & SQL API makes it possible to work with queries written in the SQL language, but these queries need to be embedded within a table program that is written in either Java or Scala. Moreover, these programs need to be packaged with a build tool before being submitted to a cluster. This more or less limits the usage of Flink to Java/Scala programmers. The SQL Client aims to provide an easy way of writing, debugging, and submitting table programs to a Flink cluster without a single line of Java or Scala code. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/models/downloads/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/models/downloads/ SQL Models download page # The page contains links to optional SQL Client models that are not part of the binary distribution. Optional SQL models # Name Version Download Link OpenAI Download (asc, sha1) //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/downloads/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/downloads/ SQL Connectors download page # The page contains links to optional SQL Client connectors and formats that are not part of the binary distribution. Optional SQL formats # Name Download link Avro Download (asc, sha1) Avro Schema Registry Download (asc, sha1) Debezium Download (asc, sha1) ORC Download (asc, sha1) Parquet Download (asc, sha1) Protobuf Download (asc, sha1) Optional SQL connectors # Name Version Download Link Amazon Kinesis Data Firehose universal Download (asc, sha1) Amazon DynamoDB universal Download (asc, sha1) Elasticsearch 6. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/memory/network_mem_tuning/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/memory/network_mem_tuning/ Network memory tuning guide # Overview # Each record in Flink is sent to the next subtask compounded with other records in a network buffer, the smallest unit for communication between subtasks. In order to maintain consistent high throughput, Flink uses network buffer queues (also known as in-flight data) on the input and output side of the transmission process. Each subtask has an input queue waiting to consume data and an output queue waiting to send data to the next subtask. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/testing/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/testing/ Testing # Testing is an integral part of every software development process as such Apache Flink comes with tooling to test your application code on multiple levels of the testing pyramid. Testing User-Defined Functions # Usually, one can assume that Flink produces correct results outside of a user-defined function. Therefore, it is recommended to test those classes that contain the main business logic with unit tests as much as possible. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/experimental/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/experimental/ Experimental Features # This section describes experimental features in the DataStream API. Experimental features are still evolving and can be either unstable, incomplete, or subject to heavy change in future versions. Reinterpreting a pre-partitioned data stream as keyed stream # We can re-interpret a pre-partitioned data stream as a keyed stream to avoid shuffling. WARNING: The re-interpreted data stream MUST already be pre-partitioned in EXACTLY the same way Flink’s keyBy would partition the data in a shuffle w. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/config/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/config/ Configuration # By default, the Table & SQL API is preconfigured for producing accurate results with acceptable performance. Depending on the requirements of a table program, it might be necessary to adjust certain parameters for optimization. For example, unbounded streaming programs may need to ensure that the required state size is capped (see streaming concepts). Overview # When instantiating a TableEnvironment, EnvironmentSettings can be used to pass the desired configuration for the current session, by passing a Configuration object to the EnvironmentSettings. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/table/metrics/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/table/metrics/ Metrics # PyFlink exposes a metric system that allows gathering and exposing metrics to external systems. Registering metrics # You can access the metric system from a Python user-defined function by calling function_context.get_metric_group() in the open method. The get_metric_group() method returns a MetricGroup object on which you can create and register new metrics. Metric types # PyFlink supports Counters, Gauges, Distribution and Meters. Counter # A Counter is used to count something. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/tuning/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/tuning/ Performance Tuning # SQL is the most widely used language for data analytics. Flink’s Table API and SQL enables users to define efficient stream analytics applications in less time and effort. Moreover, Flink Table API and SQL is effectively optimized, it integrates a lot of query optimizations and tuned operator implementations. But not all of the optimizations are enabled by default, so for some workloads, it is possible to improve performance by turning on some options. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/python_config/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/python_config/ Configuration # Depending on the requirements of a Python API program, it might be necessary to adjust certain parameters for optimization. For Python DataStream API program, the config options could be set as following: from pyflink.common import Configuration from pyflink.datastream import StreamExecutionEnvironment config = Configuration() config.set_integer("python.fn-execution.bundle.size", 1000) env = StreamExecutionEnvironment.get_execution_environment(config) For Python Table API program, all the config options available for Java/Scala Table API program could also be used in the Python Table API program. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/debugging/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/debugging/ Debugging # This page describes how to debug in PyFlink. Logging Infos # Client Side Logging # You can log contextual and debug information via print or standard Python logging modules in PyFlink jobs in places outside Python UDFs. The logging messages will be printed in the log files of the client during job submission. from pyflink.table import EnvironmentSettings, TableEnvironment # create a TableEnvironment env_settings = EnvironmentSettings.in_streaming_mode() table_env = TableEnvironment. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/table/python_table_api_connectors/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/table/python_table_api_connectors/ Connectors # This page describes how to use connectors in PyFlink and highlights the details to be aware of when using Flink connectors in Python programs. Note For general connector information and common configuration, please refer to the corresponding Java/Scala documentation. Download connector and format jars # Since Flink is a Java/Scala-based project, for both connectors and formats, implementations are available as jars that need to be specified as job dependencies. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/environment_variables/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/environment_variables/ Environment Variables # These environment variables will affect the behavior of PyFlink: Environment Variable Description FLINK_HOME PyFlink job will be compiled before submitting and it requires Flink's distribution to compile the job. PyFlink's installation package already contains Flink's distribution and it's used by default. This environment allows you to specify a custom Flink's distribution. PYFLINK_CLIENT_EXECUTABLE The path of the Python interpreter used to launch the Python process when submitting the Python jobs via " //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sourcessinks/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/sourcessinks/ User-defined Sources & Sinks # Dynamic tables are the core concept of Flink’s Table & SQL API for processing both bounded and unbounded data in a unified fashion. Because dynamic tables are only a logical concept, Flink does not own the data itself. Instead, the content of a dynamic table is stored in external systems (such as databases, key-value stores, message queues) or files. Dynamic sources and dynamic sinks can be used to read and write data from and to an external system. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/faq/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/python/faq/ FAQ # This page describes the solutions to some common questions for PyFlink users. Preparing Python Virtual Environment # You can download a convenience script to prepare a Python virtual env zip which can be used on Mac OS and most Linux distributions. You can specify the PyFlink version to generate a Python virtual environment required for the corresponding PyFlink version, otherwise the most recent version will be installed. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/java_lambdas/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/java_lambdas/ Java Lambda Expressions # Java 8 introduced several new language features designed for faster and clearer coding. With the most important feature, the so-called “Lambda Expressions”, it opened the door to functional programming. Lambda expressions allow for implementing and passing functions in a straightforward way without having to declare additional (anonymous) classes. Flink supports the usage of lambda expressions for all operators of the Java API, however, whenever a lambda expression uses Java generics you need to declare type information explicitly. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/concepts/temporal_table_function/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/table/concepts/temporal_table_function/ Temporal Table Function # A Temporal table function provides access to the version of a temporal table at a specific point in time. In order to access the data in a temporal table, one must pass a time attribute that determines the version of the table that will be returned. Flink uses the SQL syntax of table functions to provide a way to express it. Unlike a versioned table, temporal table functions can only be defined on top of append-only streams — it does not support changelog inputs. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/advanced/failure_enrichers/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/advanced/failure_enrichers/ Custom failure enrichers # Flink provides a pluggable interface for users to register their custom logic and enrich failures with extra metadata labels (string key-value pairs). This enables users to implement their own failure enrichment plugins to categorize job failures, expose custom metrics, or make calls to external notification systems. FailureEnrichers are triggered every time an exception is reported at runtime by the JobManager. Every FailureEnricher may asynchronously return labels associated with the failure that are then exposed via the JobManager’s REST API (e. //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/advanced/job_status_listener/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/advanced/job_status_listener/ Job status changed listener # Flink provides a pluggable interface for users to register their custom logic for handling with the job status changes in which lineage info about source/sink is provided. This enables users to implement their own flink lineage reporter to send lineage info to third party data lineage systems for example Datahub and Openlineage. The job status changed listeners are triggered every time status change happened for the application. //nightlies.apache.org/flink/flink-docs-release-2.2/release-notes/flink-1.10/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/release-notes/flink-1.10/ Release Notes - Flink 1.10 # These release notes discuss important aspects, such as configuration, behavior, or dependencies, that changed between Flink 1.9 and Flink 1.10. Please read these notes carefully if you are planning to upgrade your Flink version to 1.10. Clusters & Deployment # FileSystems should be loaded via Plugin Architecture # FLINK-11956 # s3-hadoop and s3-presto filesystems do no longer use class relocations and need to be loaded through plugins but now seamlessly integrate with all credential providers. //nightlies.apache.org/flink/flink-docs-release-2.2/release-notes/flink-1.11/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/release-notes/flink-1.11/ Release Notes - Flink 1.11 # These release notes discuss important aspects, such as configuration, behavior, or dependencies, that changed between Flink 1.10 and Flink 1.11. Please read these notes carefully if you are planning to upgrade your Flink version to 1.11. Clusters & Deployment # Support for Application Mode # FLIP-85 # The user can now submit applications and choose to execute their main() method on the cluster rather than the client. //nightlies.apache.org/flink/flink-docs-release-2.2/release-notes/flink-1.12/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/release-notes/flink-1.12/ Release Notes - Flink 1.12 # These release notes discuss important aspects, such as configuration, behavior, or dependencies, that changed between Flink 1.11 and Flink 1.12. Please read these notes carefully if you are planning to upgrade your Flink version to 1.12. Known Issues # Unaligned checkpoint recovery may lead to corrupted data stream # FLINK-20654 # Using unaligned checkpoints in Flink 1.12.0 combined with two/multiple inputs tasks or with union inputs for single input tasks can result in corrupted state. //nightlies.apache.org/flink/flink-docs-release-2.2/release-notes/flink-1.13/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/release-notes/flink-1.13/ Release Notes - Flink 1.13 # These release notes discuss important aspects, such as configuration, behavior, or dependencies, that changed between Flink 1.12 and Flink 1.13. Please read these notes carefully if you are planning to upgrade your Flink version to 1.13. Failover # Remove state.backend.async option. # FLINK-21935 # The state.backend.async option is deprecated. Snapshots are always asynchronous now (as they were by default before) and there is no option to configure a synchronous snapshot any more. //nightlies.apache.org/flink/flink-docs-release-2.2/release-notes/flink-1.14/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/release-notes/flink-1.14/ Release notes - Flink 1.14 # These release notes discuss important aspects, such as configuration, behavior, or dependencies, that changed between Flink 1.13 and Flink 1.14. Please read these notes carefully if you are planning to upgrade your Flink version to 1.14. Known issues # State migration issues # Some of our internal serializers i.e. RowSerializer, TwoPhaseCommitSinkFunction’s serializer, LinkedListSerializer might prevent a successful job starts if state migration is necessary. //nightlies.apache.org/flink/flink-docs-release-2.2/release-notes/flink-1.15/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/release-notes/flink-1.15/ Release notes - Flink 1.15 # These release notes discuss important aspects, such as configuration, behavior, or dependencies, that changed between Flink 1.14 and Flink 1.15. Please read these notes carefully if you are planning to upgrade your Flink version to 1.15. Summary of changed dependency names # There are Several changes in Flink 1.15 that require updating dependency names when upgrading from earlier versions, mainly including the effort to opting-out Scala dependencies from non-scala modules and reorganize table modules. //nightlies.apache.org/flink/flink-docs-release-2.2/release-notes/flink-1.16/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/release-notes/flink-1.16/ Release notes - Flink 1.16 # These release notes discuss important aspects, such as configuration, behavior, or dependencies, that changed between Flink 1.15 and Flink 1.16. Please read these notes carefully if you are planning to upgrade your Flink version to 1.16. Clusters & Deployment # Deprecate host/web-ui-port parameter of jobmanager.sh # FLINK-28735 # The host/web-ui-port parameters of the jobmanager.sh script have been deprecated. These can (and should) be specified with the corresponding options as dynamic properties. //nightlies.apache.org/flink/flink-docs-release-2.2/release-notes/flink-1.17/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/release-notes/flink-1.17/ Release notes - Flink 1.17 # These release notes discuss important aspects, such as configuration, behavior or dependencies, that changed between Flink 1.16 and Flink 1.17. Please read these notes carefully if you are planning to upgrade your Flink version to 1.17. Clusters & Deployment # Only one Zookeeper version is bundled in flink-dist # FLINK-30237 # The Flink distribution no longer bundles 2 different Zookeeper client jars (one in lib, one in lib/opt respectively). //nightlies.apache.org/flink/flink-docs-release-2.2/release-notes/flink-1.18/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/release-notes/flink-1.18/ Release notes - Flink 1.18 # These release notes discuss important aspects, such as configuration, behavior or dependencies, that changed between Flink 1.17 and Flink 1.18. Please read these notes carefully if you are planning to upgrade your Flink version to 1.18. Build System # Support Java 17 (LTS) # FLINK-15736 # Apache Flink was made ready to compile and run with Java 17 (LTS). This feature is still in beta mode. //nightlies.apache.org/flink/flink-docs-release-2.2/release-notes/flink-1.19/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/release-notes/flink-1.19/ Release notes - Flink 1.19 # These release notes discuss important aspects, such as configuration, behavior or dependencies, that changed between Flink 1.18 and Flink 1.19. Please read these notes carefully if you are planning to upgrade your Flink version to 1.19. Dependency upgrades # Drop support for python 3.7 # FLINK-33029 # Add support for python 3.11 # FLINK-33030 # Build System # Support Java 21 # FLINK-33163 # Apache Flink was made ready to compile and run with Java 21. //nightlies.apache.org/flink/flink-docs-release-2.2/release-notes/flink-1.20/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/release-notes/flink-1.20/ Release notes - Flink 1.20 # These release notes discuss important aspects, such as configuration, behavior or dependencies, that changed between Flink 1.19 and Flink 1.20. Please read these notes carefully if you are planning to upgrade your Flink version to 1.20. Checkpoints # Unified File Merging Mechanism for Checkpoints # FLINK-32070 # The unified file merging mechanism for checkpointing is introduced to Flink 1.20 as an MVP (“minimum viable product”) feature. //nightlies.apache.org/flink/flink-docs-release-2.2/release-notes/flink-1.5/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/release-notes/flink-1.5/ Release Notes - Flink 1.5 # These release notes discuss important aspects, such as configuration, behavior, or dependencies, that changed between Flink 1.4 and Flink 1.5. Please read these notes carefully if you are planning to upgrade your Flink version to 1.5. Update Configuration for Reworked Job Deployment # Flink’s reworked cluster and job deployment component improves the integration with resource managers and enables dynamic resource allocation. One result of these changes is, that you no longer have to specify the number of containers when submitting applications to YARN and Mesos. //nightlies.apache.org/flink/flink-docs-release-2.2/release-notes/flink-1.6/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/release-notes/flink-1.6/ Release Notes - Flink 1.6 # These release notes discuss important aspects, such as configuration, behavior, or dependencies, that changed between Flink 1.5 and Flink 1.6. Please read these notes carefully if you are planning to upgrade your Flink version to 1.6. Changed Configuration Default Values # The default value of the slot idle timeout slot.idle.timeout is set to the default value of the heartbeat timeout (50 s). Changed Elasticsearch 5. //nightlies.apache.org/flink/flink-docs-release-2.2/release-notes/flink-1.7/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/release-notes/flink-1.7/ Release Notes - Flink 1.7 # These release notes discuss important aspects, such as configuration, behavior, or dependencies, that changed between Flink 1.6 and Flink 1.7. Please read these notes carefully if you are planning to upgrade your Flink version to 1.7. Scala 2.12 support # When using Scala 2.12 you might have to add explicit type annotations in places where they were not required when using Scala 2.11. This is an excerpt from the TransitiveClosureNaive. //nightlies.apache.org/flink/flink-docs-release-2.2/release-notes/flink-1.8/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/release-notes/flink-1.8/ Release Notes - Flink 1.8 # These release notes discuss important aspects, such as configuration, behavior, or dependencies, that changed between Flink 1.7 and Flink 1.8. Please read these notes carefully if you are planning to upgrade your Flink version to 1.8. State # Continuous incremental cleanup of old Keyed State with TTL # We introduced TTL (time-to-live) for Keyed state in Flink 1.6 (FLINK-9510). This feature allowed to clean up and make inaccessible keyed state entries when accessing them. //nightlies.apache.org/flink/flink-docs-release-2.2/release-notes/flink-1.9/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/release-notes/flink-1.9/ Release Notes - Flink 1.9 # These release notes discuss important aspects, such as configuration, behavior, or dependencies, that changed between Flink 1.8 and Flink 1.9. It also provides an overview on known shortcoming or limitations with new experimental features introduced in 1.9. Please read these notes carefully if you are planning to upgrade your Flink version to 1.9. Known shortcomings or limitations for new features # New Table / SQL Blink planner # Flink 1. //nightlies.apache.org/flink/flink-docs-release-2.2/release-notes/flink-2.0/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/release-notes/flink-2.0/ Release notes - Flink 2.0 # These release notes discuss important aspects, such as configuration, behavior or dependencies, that changed between Flink 1.20 and Flink 2.0. Please read these notes carefully if you are planning to upgrade your Flink version to 2.0. New Features & Behavior Changes # State & Checkpoints # Disaggregated State Storage and Management # FLINK-32070 # The past decade has witnessed a dramatic shift in Flink’s deployment mode, workload patterns, and hardware improvements. //nightlies.apache.org/flink/flink-docs-release-2.2/release-notes/flink-2.1/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/release-notes/flink-2.1/ Release notes - Flink 2.1 # These release notes discuss important aspects, such as configuration, behavior or dependencies, that changed between Flink 2.0 and Flink 2.1. Please read these notes carefully if you are planning to upgrade your Flink version to 2.1. Table SQL / API # Model DDLs using Table API # FLINK-37548 # Since Flink 2.0, we have introduced dedicated syntax for AI models, enabling users to define models as easily as creating catalog objects and invoke them like standard functions or table functions in SQL statements. //nightlies.apache.org/flink/flink-docs-release-2.2/release-notes/flink-2.2/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/release-notes/flink-2.2/ Release notes - Flink 2.2 # These release notes discuss important aspects, such as configuration, behavior or dependencies, that changed between Flink 2.1 and Flink 2.2. Please read these notes carefully if you are planning to upgrade your Flink version to 2.2. Table SQL / API # Support VECTOR_SEARCH in Flink SQL # FLINK-38422 # Apache Flink has supported leveraging LLM capabilities through the ML_PREDICT function in Flink SQL since version 2. //nightlies.apache.org/flink/flink-docs-release-2.2/versions/ Mon, 01 Jan 0001 00:00:00 +0000 //nightlies.apache.org/flink/flink-docs-release-2.2/versions/ Versions # An appendix of hosted documentation for all versions of Apache Flink. v2.2 v2.1 v2.0 v1.20 v1.19 v1.18 v1.17 v1.16 v1.15 v1.14 v1.13 v1.12 v1.11 v1.10 v1.9 v1.8 v1.7 v1.6 v1.5 v1.4 v1.3 v1.2 v1.1 v1.0