morling.dev -- Blog

Improved Column Reader API, First Cut of Geospatial Support: Hardwood 1.0.0.CR1 Is Available

Sun, 31 May 2026 21:36:00 +0100

Table of Contents

I am happy to announce the release of Hardwood 1.0.0.CR1!

This first candidate release of Hardwood 1.0 brings a substantially improved API for columnar access to Apache Parquet files, initial support for Parquet’s GEOMETRY/GEOGRAPHY column types, and many other improvements to the core library as well as the Hardwood CLI.

VARIANT Support, Interactive Parquet File TUI: Hardwood 1.0.0.Beta2 Is Out

Wed, 29 Apr 2026 18:45:00 +0100

Table of Contents

I am happy to announce the release of Hardwood 1.0.0.Beta2!

The latest version of this new parser for Apache Parquet comes with support for VARIANT columns, an interactive text-based UI (TUI) for examining and analysing the structure of Parquet files, significantly improved performance, more efficient reading of files from object storage, and much more.

Hardwood Reaches Beta: S3, Predicate Push-Down, CLI, and More

Thu, 02 Apr 2026 19:20:00 +0100

Table of Contents

I am pleased to announce the release of Hardwood 1.0.0.Beta1!

Hardwood is a new parser for Apache Parquet, optimized for minimal dependencies and great performance. Since the project’s initial release just a few weeks back, a small yet very active community has come together and evolved Hardwood significantly. Today, we are shipping an S3 backend, allowing to parse files directly from object storage, predicate pushdown for both local and remote files, Avro bindings, a CLI for inspecting Parquet files, and much more. We’re also excited to launch a website for the project, hardwood.dev, which contains the documentation and API reference.

Let’s dig in.

Hardwood: A New Parser for Apache Parquet

Thu, 26 Feb 2026 14:30:00 +0200

Table of Contents

Today, it’s my great pleasure to announce the first public release of Hardwood, a new parser for the Apache Parquet file format, optimized for minimal dependencies and great performance.

Hardwood is open-source (Apache License 2.0) and supports Java 21 or newer. You can grab it from Maven Central and start parsing your Parquet files with ease and efficiency.

You Gotta Push If You Wanna Pull

Sun, 07 Dec 2025 10:05:00 +0100

Table of Contents

Historically, data management systems have been built around the notion of pull queries: users query data which, for instance, is stored in tables in an RDBMS, Parquet files in a data lake, or a full-text index in Elasticsearch. When a user issues a query, the engine will produce the result set at that point in time by churning through the data set and finding all matching records (oftentimes sped up by utilizing indexes).

On Idempotency Keys

Tue, 25 Nov 2025 13:10:00 +0100

Table of Contents

In distributed systems, there’s a common understanding that it is not possible to guarantee exactly-once delivery of messages. What is possible though is exactly-once processing. By adding a unique idempotency key to each message, you can enable consumers to recognize and ignore duplicate messages, i.e. messages which they have received and successfully processed before.

Building a Durable Execution Engine With SQLite

Thu, 20 Nov 2025 13:10:00 +0100

Table of Contents

Lately, there has been a lot of excitement around Durable Execution (DE) engines. The basic idea of DE is to take (potentially long-running) multi-step workflows, such as processing a purchase order or a user sign-up, and make their individual steps persistent. If a flow gets interrupted while running, for instance due to a machine failure, the DE engine can resume it from the last successfully executed step and drive it to completion.

"You Don't Need Kafka, Just Use Postgres" Considered Harmful

Mon, 03 Nov 2025 18:02:00 +0100

Looking to make it to the front page of HackerNews? Then writing a post arguing that "Postgres is enough", or why "you don’t need Kafka at your scale" is a pretty failsafe way of achieving exactly that. No matter how often it has been discussed before, this topic is always doing well. And sure, what’s not to love about that? I mean, it has it all: Postgres, everybody’s most favorite RDBMS—check! Keeping things lean and easy—sure, count me in! A somewhat spicy take—bring it on!

Let's Take a Look at... Lower Java Tail Latencies With ZGC

Wed, 17 Sep 2025 17:09:00 +0200

Table of Contents

In the "Let’s Take a Look at…!" blog series I am exploring interesting projects, developments and technologies in the data and streaming space. This can be KIPs and FLIPs, open-source projects, services, relevant improvements to Java and the JVM, and more. The idea is to get some hands-on experience, learn about potential use cases and applications, and understand the trade-offs involved. If you think there’s a specific subject I should take a look at, let me know in the comments below.

Java 25 was released earlier this week, and it is the first Java release with long-term support (LTS) which ships with Generational ZGC as the one (and only) flavor of the ZGC garbage collector. ZGC itself is a relatively new concurrent collector, originally added in Java 11.

Postgres Replication Slots: Confirmed Flush LSN vs. Restart LSN

Tue, 05 Aug 2025 13:55:00 +0200

Table of Contents

Replication slots in Postgres keep track of how far consumers have read a replication stream. After a restart, consumers—either Postgres read replicas or external tools for change data capture (CDC), like Debezium—resume reading from the last confirmed log sequence number (LSN) of their replication slot. The slot prevents the database from disposing of required log segments, allowing safe resumption after downtime.

In this post, we are going to take a look at why Postgres replication slots don’t have one but two LSN-related attributes: restart_lsn and confirmed_flush_lsn. Understanding the difference between the two is crucial for troubleshooting replication issues, optimizing WAL retention, and avoiding common pitfalls in production environments.

Converting Future to CompletableFuture With Java Virtual Threads

Thu, 17 Jul 2025 10:25:00 +0200

This post explores how virtual threads in Java 21+ provide an elegant solution for converting legacy Future objects into CompletableFuture instances.

Since Java 8, the CompletableFuture API provides a convenient way for performing asynchronous operations in a functional, composable way. This makes it very simple to call some long-running methods—for instance involving external I/O—asynchronously and process each result as soon as it is available, without blocking on any threads:

Mastering Postgres Replication Slots: Preventing WAL Bloat and Other Production Issues

Tue, 08 Jul 2025 13:55:00 +0200

Table of Contents

Over the last couple of years, I’ve helped dozens of users and organizations to build Change Data Capture (CDC) pipelines for their Postgres databases. A key concern in that process is setting up and managing replication slots, which are Postgres' mechanism for making sure that any segments of the write-ahead log (WAL) of the database are kept around until they have been processed by registered replication consumers.

When not being careful, a replication slot may cause unduly large amounts of WAL segments to be retained by the database. This post describes best practices helping to prevent this and other issues, discussing aspects like heartbeats, replication slot failover, monitoring, the management of Postgres publications, and more. While this is primarily based on my experience of using replication slots via Debezium’s Postgres connector, the principles are generally applicable and are worth considering also when using other CDC tools for Postgres based on logical replication.

This AI Agent Should Have Been a SQL Query

Wed, 18 Jun 2025 15:30:00 +0200

Table of Contents

AI Agents have improved in leaps and bounds in recent times, moving beyond simple chatbots to sophisticated, autonomous systems. This post explores a novel approach to building agentic systems: using the power of streaming SQL queries. Discover how platforms like Apache Flink can transform the development of AI Agents, offering benefits in consistency, scalability, and developer experience.

Backfilling Postgres TOAST Columns in Debezium Data Change Events

Mon, 26 May 2025 16:40:00 +0200

Table of Contents

Postgres logical replication, while powerful for capturing real-time data changes, presents challenges with TOAST columns, whose values can be absent from data change events in specific situations. This post discusses how Debezium addresses this through its built-in reselect post processor, then explores more robust solutions leveraging Apache Flink’s capabilities for stateful stream processing, including Flink SQL and the brand-new process table functions (PTFs) in Flink 2.1.

"Streaming vs. Batch" Is a Wrong Dichotomy, and I Think It's Confusing

Wed, 14 May 2025 10:10:00 +0200

Often times, "Stream vs. Batch" is discussed as if it’s one or the other, but to me this does not make that much sense really.

What If We Could Rebuild Kafka From Scratch?

Thu, 24 Apr 2025 16:25:00 +0200

The last few days I spent some time digging into the recently announced KIP-1150 ("Diskless Kafka"), as well AutoMQ’s Kafka fork, tightly integrating Apache Kafka and object storage, such as S3. Following the example set by WarpStream, these projects aim to substantially improve the experience of using Kafka in cloud environments, providing better elasticity, drastically reducing cost, and paving the way towards native lakehouse integration.

This got me thinking, if we were to start all over and develop a durable cloud-native event log from scratch—Kafka.next if you will—which traits and characteristics would be desirable for this to have? Separating storage and compute and object store support would be table stakes, but what else should be there? Having used Kafka for many years for building event-driven applications as well as for running realtime ETL and change data capture pipelines, here’s my personal wishlist:

A Deep Dive Into Ingesting Debezium Events From Kafka With Flink SQL

Wed, 16 Apr 2025 11:25:00 +0200

Table of Contents

Over the years, I’ve spoken quite a bit about the use cases for processing Debezium data change events with Apache Flink, such as metadata enrichment, building denormalized data views, and creating data contracts for your CDC streams. One detail I haven’t covered in depth so far is how to actually ingest Debezium change events from a Kafka topic into Flink, in particular via Flink SQL. Several connectors and data formats exist for this, which can make things somewhat confusing at first. So let’s dive into the different options and the considerations around them!

Building a Native Binary for Apache Kafka on macOS

Mon, 07 Apr 2025 12:25:00 +0200

Table of Contents

KIP-974: Docker Image for GraalVM based Native Kafka Broker

With help of the GraalVM configuration developed for KIP-974 (Docker Image for GraalVM based Native Kafka Broker), you can easily build a self-contained native binary for Apache Kafka. Read on to learn how you can build a native Kafka executable yourself, starting in milli-seconds, making it a perfect fit for development and testing purposes.

When I wrote about ahead-of-time class loading and linking in Java 24 recently, I also published the start-up time for Apache Kafka as a native binary for comparison. This was done via Docker, as there’s no pre-built native binary of Kafka available for the operating system I’m running on, macOS. But there is a native Kafka container image, so this is what I chose for the sake of convenience.

Now, running in a container adds a little bit of overhead of course, so it wasn’t a surprise when Thomas Würthinger, lead of the GraalVM project at Oracle, brought up the question what the value would be when running Kafka natively on macOS. Needless to say I can’t leave this kind of nice nerd snipe pass, so I set out to learn how to build a native Kafka binary on macOS, using GraalVM.

Let's Take a Look at... JEP 483: Ahead-of-Time Class Loading & Linking!

Thu, 27 Mar 2025 14:00:00 +0100

Table of Contents

Java 24 got released last week, and what a meaty release it is: more than twenty Java Enhancement Proposals (JEPs) have been shipped, including highlights such as compact object headers (JEP 450, I hope to spend some time diving into that one some time soon), a new class-file API (JEP 484), and more flexible constructor bodies (JEP 492, third preview). One other JEP which might fly a bit under the radar is JEP 483 ("Ahead-of-Time Class Loading & Linking"). It promises to reduce the start-up time of Java applications without requiring any modifications to the application itself, what’s not to be liked about that? Let’s take a closer look!

The Synchrony Budget

Tue, 18 Mar 2025 14:00:00 +0100

For building a system of distributed services, one concept I think is very valuable to keep in mind is what I call the synchrony budget: as much as possible, a service should minimize the number of synchronous requests which it makes to other services.

Let's Take a Look at... KIP-932: Queues for Kafka!

Wed, 05 Mar 2025 12:35:00 +0100

Table of Contents

In the "Let’s Take a Look at…!" blog series I am going to explore interesting projects, developments and technologies in the data and streaming space. This can be KIPs and FLIPs, open-source projects, services, and more. The idea is to get some hands-on experience, learn about potential use cases and applications, and understand the trade-offs involved. If you think there’s a specific subject I should take a look at, let me know in the comments below!

That guy above? Yep, that’s me, whenever someone says "Kafka queue". Because, that’s not what Apache Kafka is. At its core, Kafka is a distributed durable event log. Producers write events to a topic, organized in partitions which are distributed amongst the brokers of a Kafka cluster. Consumers, organized in groups, divide the partitions they process amongst themselves, so that each partition of a topic is read by exactly one consumer in the group.

Get Running with Apache Flink on Kubernetes, part 2 of 2

Tue, 28 Jan 2025 00:00:00 +0000

Table of Contents

Welcome back to this two-part blog post series about running Apache Flink on Kubernetes, using the Flink Kubernetes operator. In part one, we discussed installation and setup of the operator, different deployment types, how to deploy Flink jobs using custom Kubernetes resources, and how to create container images for your own Flink jobs. In this part, we’ll focus on aspects such as fault tolerance and high availability of your Flink jobs running on Kubernetes, savepoint management, observability, and more. You can find the complete source code for all the examples shown in this series in the Decodable examples repository on GitHub: on GitHub.

Get Running with Apache Flink on Kubernetes, part 1 of 2

Tue, 21 Jan 2025 00:00:00 +0000

Table of Contents

Kubernetes is a widely used deployment platform for Apache Flink. While Flink has had native support for Kubernetes for quite a while, it is in particular the operator pattern which makes deploying Flink jobs onto Kubernetes clusters a compelling option: you define jobs in a declarative resource, and a control loop running in a component called a Kubernetes operator takes care of provisioning and maintaining (e.g. scaling, updating) all the required resources. Automation is the keyword here, significantly reducing the manual effort required for running Flink jobs in production.

Failover Replication Slots with Postgres 17

Tue, 03 Dec 2024 00:00:00 +0000

Table of Contents

Postgres read replicas are commonly used not only to distribute query load amongst multiple nodes, but also to ensure high availability (HA) of the database. If the primary node of a Postgres cluster fails, a read replica can be promoted to be the new primary, processing write (and read) requests from thereon.

Thoughts On Moving Debezium to the Commonhaus Foundation

Wed, 27 Nov 2024 17:25:00 +0100

If you are following the news around Debezium—an open-source platform for Change Data Capture (CDC) for a variety of databases—you may have seen the announcement that the project is in the process of moving to the Commonhaus Foundation. I think this is excellent news for the Debezium project, its community, and open-source CDC at large. In this post I’d like to share some more context on why I am so excited about this development.

Building OpenJDK From Source On macOS

Sat, 16 Nov 2024 15:25:00 +0100

Every now and then, it can come in very handy to build OpenJDK from source yourself, for instance if you want to explore a feature which is under development on a branch for which no builds are published. For some reason I always thought that building OpenJDK is a very complex processing, requiring the installation of arcane tool chains etc. But as it turns out, this actually not true: the project does a great job of documenting what’s needed and only a few steps are necessary to build your very own JDK.

Revisiting the Outbox Pattern

Thu, 31 Oct 2024 00:00:00 +0000

Over the last few years, the outbox pattern has become a common solution for implementing data exchange flows between microservices. It allows services to safely and reliably update their own local datastore and at the same time send out notifications to other services via data streaming platforms such as Apache Kafka. But time isn’t standing still: people ask about disadvantages of the pattern (is the database becoming a bottleneck?), alternative solutions such as "listen-to-yourself" have been proposed, Kafka is about to get support for participating in 2-phase commit (2PC) transactions, and more. It’s time to take another look at the outbox pattern, what it is and how to implement it, whether it’s still relevant in 2024, and which alternatives exist!

CDC Is a Feature Not a Product

Fri, 18 Oct 2024 15:55:00 +0200

During and after my time as the lead of Debezium, a widely used open-source platform for Change Data Capture (CDC) for a variety of database, I got repeatedly asked whether I’d be interested in creating a company around CDC. VCs, including wellknown household names, did and do reach out to me, pitching this idea.

How I Am Setting Up VMs On Hetzner Cloud

Sun, 06 Oct 2024 20:25:00 +0200

Table of Contents

Whenever I’ve need a Linux box for some testing or experimentation, or projects like the One Billion Row Challenge a few months back, my go-to solution is Hetzner Online, a data center operator here in Europe.

Their prices for VMs are unbeatable, starting with 3,92 €/month for two shared vCPUs (either x64 or AArch64), four GB of RAM, and 20 TB of network traffic (these are prices for their German data centers, they vary between regions). four dedicated cores with 16 GB, e.g. for running a small web server, will cost you 28.55 €/month. Getting a box with similar specs on AWS would set you back a multiple of that, with the (outbound) network cost being the largest chunk. So it’s not a big surprise that more and more people realize the advantages of this offering, most notably Ruby on Rails creator David Heinemeier Hansson, who has been singing the praise for Hetzner’s dedicated servers, but also their VM instances, quite a bit on Twitter lately.

Leader Election With S3 Conditional Writes

Mon, 26 Aug 2024 09:15:00 +0100

Table of Contents

In distributed systems, for instance when scaling out some workload to multiple compute nodes, it is a common requirement to select a leader for performing a given task: only one of the nodes should process the records from a Kafka topic partition, write to a file system, call a remote API, etc. Otherwise, multiple workers may end up doing the same task twice, overwriting each other’s data, and worse.

Shell Spell: Extracting and Propagating Multiple Values With jq

Sat, 06 Jul 2024 11:20:00 +0200

In my day job at Decodable, I am currently working with Terraform to provision some cloud infrastructure for an upcoming hands-on lab. Part of this set-up is a Postgres database on Amazon RDS, which I am creating using the Terraform AWS modules. Now, once my database was up and running, I wanted to extract two dynamically generated values from Terraform: the random password created for the root user, and the database host URL. On my way down the rabbit hole for finding a CLI command for doing this efficiently, I learned a few interesting shell details which I’d like to share.

A Zipping Gatherer

Mon, 18 Mar 2024 18:12:00 +0100

The other day, I was looking for means of zipping two Java streams: connecting them element by element—essentially a join based on stream offset position—and emitting an output stream with the results. Unfortunately, there is no zip() method offered by the Java Streams API itself. While it was considered for inclusion in early preview versions, the method was removed before the API went GA with Java 8 and you have to resort to 3rd party libraries such as Google Guava if you need this functionality.

A Taxonomy Of Data Change Events

Wed, 13 Mar 2024 00:00:00 +0000

Table of Contents

Data change events are at the core of Change Data Capture (CDC) solutions such as Debezium. They describe the changes made to a specific record in a database and allow event consumers to take action based on this information, enabling a wide range of use cases, such as real-time ETL (by propagating the updated data into downstream data stores such as data warehouses, analytics databases, or fulltext search indexes), microservices data exchange, or audit logging.

Last Updated Columns With Postgres

Tue, 20 Feb 2024 09:15:00 +0100

In many applications it’s a requirement to keep track of when a record was created and updated the last time. Often, this is implemented by having columns such as created_at and updated_at within each table. To make things as simple as possible for application developers, the database itself should take care of maintaining these values automatically when a record gets inserted or updated.

Filtering Process Output With tee

Sat, 10 Feb 2024 11:11:00 +0100

Recently I ran into a situation where it was necessary to capture the output of a Java process on the stdout stream, and at the same time a filtered subset of the output in a log file. The former, so that the output gets picked up by the Kubernetes logging infrastructure. The letter for further processing on our end: we were looking to detect when the JVM stops due to an OutOfMemoryError, passing on that information to some error classifier.

1BRC—The Results Are In!

Sun, 04 Feb 2024 16:34:00 +0100

Table of Contents

Oh what a wild ride the last few weeks have been. The One Billion Row Challenge (1BRC for short), something I had expected to be interesting to a dozen folks or so at best, has gone kinda viral, with hundreds of people competing and engaging. In Java, as intended, but also beyond: folks implemented the challenge in languages such as Go, Rust, C/C++, C#, Fortran, or Erlang, as well databases (Postgres, Oracle, Snowflake, etc.), and tools like awk.

It’s really incredible how far people have pushed the limits here. Pull request by pull request, the execution times for solving the problem layed out in the challenge — aggregating random temperature values from a file with 1,000,000,000 rows — improved by two orders of magnitudes in comparison to the initial baseline implementation. Today I am happy to share the final results, as the challenge closed for new entries after exactly one month on Jan 31 and all submissions have been reviewed.

The One Billion Row Challenge

Mon, 01 Jan 2024 00:00:00 +0100

Update Jan 4: Wow, this thing really took off! 1BRC is discussed at a couple of places on the internet, including Hacker News, lobste.rs, and Reddit.

For folks to show-case non-Java solutions, there is a "Show & Tell" now, check that one out for 1BRC implementations in Rust, Go, C++, and others. Some interesting related write-ups include 1BRC in SQL with DuckDB by Robin Moffatt and 1 billion rows challenge in PostgreSQL and ClickHouse by Francesco Tisiot.

Thanks a lot for all the submissions, this is going way beyond what I’d have expected! I am behind a bit with evalutions due to the sheer amount of entries, I will work through them bit by bit. I have also made a few clarifications to the rules of the challenge; please make sure to read them before submitting any entries.

Let’s kick off 2024 true coder style—I’m excited to announce the One Billion Row Challenge (1BRC), running from Jan 1 until Jan 31.

Your mission, should you decide to accept it, is deceptively simple: write a Java program for retrieving temperature measurement values from a text file and calculating the min, mean, and max temperature per weather station. There’s just one caveat: the file has 1,000,000,000 rows!

Logical Replication From Postgres 16 Stand-By Servers—Debezium and Failover Slots

Tue, 19 Dec 2023 00:00:00 +0000

Table of Contents

Welcome back to this series about logical replication from Postgres 16 stand-by servers, in which we’ll discuss how to use this feature with Debezium—a popular open-source platform for Change Data Capture (CDC) for a wide range of databases—as well as how to manage logical replication in case of failover scenarios, i.e. a situation where your primary Postgres server becomes unavailable and a stand-by server needs to take over.

Using Stand-by Servers for Postgres Logical Replication

Tue, 19 Dec 2023 00:00:00 +0000

Table of Contents

For users of Change Data Capture (CDC), one of the most exciting features in Postgres version 16 (released in September this year) is the support for logical replication from stand-by servers. Instead of connecting to your primary server, you can now point CDC tools such as Debezium to a replica server, which is very interesting for instance from a load distribution perspective. I am going to take a closer look at this new feature in this two-part blog series:

Tracking Java Native Memory With JDK Flight Recorder

Sun, 17 Dec 2023 19:29:00 +0100

Table of Contents

As regular readers of this blog will now, JDK Flight Recorder (JFR) is one of my favorite tools of the Java platform. This low-overhead event recording engine built into the JVM is invaluable for observing the runtime characteristics of Java applications and identifying any potential performance issues. JFR continues to become better and better with every new release, with one recent addition being support for native memory tracking (NMT).

Getting Started With PyFlink on Kubernetes

Thu, 07 Dec 2023 00:00:00 +0000

Table of Contents

The other day, I wanted to get my feet wet with PyFlink. While there is a fair amount of related information out there, I couldn’t find really up-to-date documentation on using current versions of PyFlink with Flink on Kubernetes.

"Change Data Capture Breaks Encapsulation". Does it, though?

Tue, 21 Nov 2023 00:00:00 +0000

Table of Contents

Having worked on Debezium—an open-source platform for Change Data Capture (CDC)—for several years, one concern I’ve heard repeatedly is this: aren’t you breaking the encapsulation of your application when you expose change event feeds directly from your database? After all, CDC exposes your internal persistent data model to the outside world, which may have unintended consequences, e.g. in terms of data exposure but also when it comes to changes to the schema of your data, which may break downstream consumers.

Can Debezium Lose Events?

Tue, 14 Nov 2023 15:00:00 +0100

This question came up on the Data Engineering sub-reddit the other day: Can Debezium lose any events? I.e. can there be a situation where a record in a database get inserted, updated, or deleted, but Debezium fails to capture that event from the transaction log and propagate it to downstream consumers?

CDC Use Cases: 7 Ways to Put CDC to Work

Thu, 02 Nov 2023 00:00:00 +0000

Table of Contents

Change Data Capture (CDC) is a powerful tool in data engineering and has seen a tremendous uptake in organizations of all kinds over the last few years. This is because it enables the tight integration of transactional databases into many other systems in your business at a very low latency.

Finding Java Thread Leaks With JDK Flight Recorder and a Bit Of SQL

Tue, 28 Feb 2023 22:16:10 +0100

The other day at work, we had a situation where we suspected a thread leak in one particular service, i.e. code which continuously starts new threads, without taking care of ever stopping them again. Each thread requires a bit of memory for its stack space, so starting an unbounded number of threads can be considered as a form of memory leak, causing your application to run out of memory eventually. In addition, the more threads there are, the more overhead the operating system incurs for scheduling them, until the scheduler itself will consume most of the available CPU resources. Thus it’s vital to detect and fix this kind of problem early on.

Getting Started With Java Development in 2023 — An Opinionated Guide

Sun, 15 Jan 2023 22:10:00 +0100

Table of Contents

27 years of age, and alive and kicking — The Java platform regularly comes out amongst the top contenders in rankings like the TIOBE index. In my opinion, rightly so. The language is very actively maintained and constantly improved; its underlying runtime, the Java Virtual Machine (JVM), is one of, if not the most, advanced runtime environments for managed programming languages.

There is a massive eco-system of Java libraries which make it a great tool for a large number of use cases, ranging from command-line and desktop applications, over web apps and backend web services, to datastores and stream processing platforms. With upcoming features like support for vectorized computations (SIMD), light-weight virtual threads, improved integration with native code, value objects and user-defined primitives, and others, Java is becoming an excellent tool for solving a larger number of software development tasks than ever before.

Oh... This is Prod?!

Thu, 05 Jan 2023 22:10:00 +0100

I strongly believe that you should avoid connecting to production environments from local developer machines as much as possible. But sometimes, e.g. in order to analyse some specific kinds of failures, doing so can be inevitable.

Now, if this is the case, I really, really want to be sure that I’m aware of the environment I am working in. I absolutely want to avoid a situation as in the catchy title of this post, when for instance you realize that you just ran some integration test against a production environment. In the context of working with the AWS CLI tool this means I’d like to be aware of the currently active profile by means of coloring my shell accordingly. Here’s how I’ve set this up using iTerm2 and zsh.

Is your Blocking Queue... Blocking?

Tue, 03 Jan 2023 22:10:00 +0100

Table of Contents

Java’s BlockingQueue hierarchy is widely used for coordinating work between different producer and consumer threads. When set up with a maximum capacity (i.e. a bounded queue), no more elements can be added by producers to the queue once it is full, until a consumer has taken at least one element. For scenarios where new work may arrive more quickly than it can be consumed, this applies means of back-pressure, ensuring the application doesn’t run out of memory eventually, while enqueuing more and more work items.

Maven, What Are You Waiting For?!

Sun, 18 Dec 2022 13:45:00 +0100

As part of my new job at Decodable, I am also planning to contribute to the Apache Flink project (as Decodable’s fully-managed stream processing platform is based on Flink). Right now, I am in the process of familiarizing myself with the Flink code base, and as such I am of course building the project from source, too.

Postgres 15: Logical Decoding Row Filters With Debezium

Thu, 15 Dec 2022 00:00:00 +0000

Table of Contents

Since logical decoding was added to Postgres in version 9.4, this powerful feature for capturing changes from the write-ahead log of the database has been continuously improved. Postgres 15, released in October this year, added support for fine-grained control over which columns (by means of column lists) and rows (via row filters) should be exported from captured tables. This means, in relational terminology, projections and filters are now natively supported by Postgres change event publications.

The Insatiable Postgres Replication Slot

Wed, 30 Nov 2022 14:00:00 +0100

Table of Contents

While working on a demo for processing change events from Postgres with Apache Flink, I noticed an interesting phenomenon: A Postgres database which I had set up for that demo on Amazon RDS, ran out of disk space. The machine had a disk size of 200 GiB which was fully used up in the course of less than two weeks.

Now a common cause for this kind of issue are replication slots which are not advanced: in that case, Postgres will hold on to all WAL segments after the latest log sequence number (LSN) which was confirmed for that slot. Indeed I had set up a replication slot (via the Decodable CDC source connector for Postgres, which is based on Debezium). I then had stopped that connector, causing the slot to become inactive. The problem was though that I was really sure that there was no traffic in that database whatsoever! What could cause a WAL growth of ~18 GB/day then?

Running a Quarkus Native Application on Render

Mon, 28 Nov 2022 11:45:00 +0100

Table of Contents

This is a quick run down of the steps required for running JVM applications, built using Quarkus and GraalVM, on Render.

Render is a cloud platform for running websites and applications. Like most other comparable services such as fly.io, it offers a decent free tier, which lets you try out the service without any financial commitment. Unlike most other services, with Render, you don’t need to provide a credit card in order to use the free tier. Which means there’s no risk of surprise bills, as often is the case with pay-per-use models, where a malicious actor could DDOS your service and drive up cost for consumed CPU resources or egress bandwidth indefinitely.

Why I Joined Decodable

Thu, 03 Nov 2022 15:00:00 +0100

Table of Contents The Space: Real-Time Stream Processing The Environment: A Start-up The Team: One of a Kind Outlook It’s my first week as a software engineer at Decodable, a start-up building a serverless real-time data platform! When I shared this news on social media yesterday, folks were not only super supportive and excited for me (thank you so much for all the nice words and wishes!), but some also asked about the reasons behind my decision for switching jobs and going to a start-up, after having worked for Red Hat for the last few years.

An Ideation for Kubernetes-native Kafka Connect

Tue, 06 Sep 2022 14:20:00 +0100

Table of Contents Standalone or Distributed? Issues with Kafka Connect on Kubernetes A Vision for Kubernetes-native Kafka Connect Kafka Connect, part of the Apache Kafka project, is a development framework and runtime for connectors which either ingest data into Kafka clusters (source connectors) or propagate data from Kafka into external systems (sink connectors). A diverse ecosystem of ready-made connectors has come to life on top of Kafka Connect, which lets you connect all kinds of data stores, APIs, and other systems to Kafka in a no-code approach.

Testing Kafka Connectors

Thu, 25 Aug 2022 09:20:00 +0100

Table of Contents

Kafka Connect is a key factor for the wide-spread adoption of Apache Kafka: a framework and runtime environment for connectors, it makes the task of getting data either into Kafka or out of Kafka solely a matter of configuration, rather than a bespoke programming job. There’s dozens, if not hundreds, of readymade source and sink connectors, allowing you to create no-code data pipelines between all kinds of databases, APIs, and other systems.

There may be situations though where there is no existing connector matching your requirements, in which case you can implement your own custom connector using the Kafka Connect framework. Naturally, this raises the question of how to test such a Kafka connector, making sure it propagates the data between the connected external system and Kafka correctly and completely. In this blog post I’d like to focus on testing approaches for Kafka Connect source connectors, i.e. connectors like Debezium, which ingest data from an external system into Kafka. Very similar strategies can be employed for testing sink connectors, though.

Ten Tips to Make Conference Talks Suck Less

Thu, 23 Jun 2022 10:15:00 +0100

Table of Contents 1. 💦 Rehearse, Rehearse, Rehearse 2. 🎬 Start With a Mission 3. 📖 Tell a Story 4. 👀 Look at the Audience, Not Your Slides 5. 🧹 Put Less Text on Your Slides. Much Less 6. ✂️ Tailor the Talk Towards Your Audience 7. 3️⃣ Rule of Three 8. 🚑 Have a Fallback Plan for Demos 9. 💪 Play to Your Strengths 10. 🔄 Circle Back Every so often, I come across some conference talk which is highly interesting in terms of its actual contents, but which unfortunately is presented in a less than ideal way.

Loom and Thread Fairness

Fri, 27 May 2022 09:20:00 +0100

Table of Contents

Project Loom (JEP 425) is probably amongst the most awaited feature additions to Java ever; its implementation of virtual threads (or "green threads") promises developers the ability to create highly concurrent applications, for instance with hundreds of thousands of open HTTP connections, sticking to the well-known thread-per-request programming model, without having to resort to less familiar and often more complex to use reactive approaches.

Having been in the workings for several years, Loom got merged into the mainline of OpenJDK just recently and is available as a preview feature in the latest Java 19 early access builds. I.e. it’s the perfect time to get your hands onto virtual threads and explore the new feature. In this post I’m going to share an interesting aspect I learned about thread scheduling fairness for CPU-bound workloads running on Loom.

Running JDK Mission Control on Apple M1

Tue, 17 May 2022 09:20:00 +0100

JDK Mission Control (JMC) is invaluable for analysing performance data recording using JDK Flight Recorder (JFR). The other day, I ran into a problem when trying to run JMC on my Mac Mini M1. Mostly for my own reference, here’s what I did to overcome it.

The Code Review Pyramid

Thu, 10 Mar 2022 16:45:00 +0100

Table of Contents FAQ When it comes to code reviews, it’s a common phenomenon that there is much focus and long-winded discussions around mundane aspects like code formatting and style, whereas important aspects (does the code change do what it is supposed to do, is it performant, is it backwards-compatible for existing clients, and many others) tend to get less attention. To raise awareness for the issue and providing some guidance on aspects to focus on, I shared a small visual on Twitter the other day, which I called the "Code Review Pyramid".

The JDK Flight Recorder File Format

Sun, 20 Feb 2022 21:15:00 +0100

The JDK Flight Recorder (JFR) is one of Java’s secret weapons; deeply integrated into the Hotspot VM, it’s a high-performance event collection framework, which lets you collect metrics on runtime aspects like object allocation and garbage collection, class loading, file and network I/O, and lock contention, do method profiling, and much more.

JFR data is persisted in recording files (since Java 14, also "realtime" event streaming is supported), which can be loaded for analysis into tools like JDK Mission Control (JMC), or the jfr utility coming with OpenJDK itself.

What's in a Good Error Message?

Wed, 12 Jan 2022 19:20:00 +0100

Table of Contents

As software developers, we’ve all come across those annoying, not-so-useful error messages when using some library or framework: "Couldn’t parse config file", "Lacking permission for this operation", etc. Ok, ok, so something went wrong apparently; but what exactly? What config file? Which permissions? And what should you do about it? Error messages lacking this kind of information quickly create a feeling of frustration and helplessness.

So what makes a good error message then? To me, it boils down to three pieces of information which should be conveyed by an error message:

Context: What led to the error? What was the code trying to do when it failed?
The error itself: What exactly failed?
Mitigation: What needs to be done in order to overcome the error?

Announcing the First Release of kcctl

Tue, 21 Dec 2021 17:48:00 +0100

Table of Contents

Discussion and Outlook

🧸 It’s Casey. Casey Cuddle.

I am very happy to announce the first stable release of kcctl, a modern and intuitive command line client for Apache Kafka Connect!

Forget about having to memorize and type the right REST API paths and curl flags; with kcctl, managing your Kafka connectors is done via concise and logically structured commands, modeled after the semantics of the kubectl tool known from Kubernetes.

Introducing the OSS Quickstart Archetype

Thu, 02 Dec 2021 18:30:00 +0100

Table of Contents

I am very happy to announce the availability of the OSS Quickstart Archetype!

Part of the ModiTect family of open-source projects, this is a Maven archetype which makes it very easy to bootstrap new Maven-based open-source projects, satisfying common requirements such as configuring plug-in versions, and adhering to best practices like auto-formatting the source code. Think Maven Quickstart Archetype and friends, but more modern, complete, and opinionated.

O Kafka, Where Art Thou?

Mon, 29 Nov 2021 18:55:00 +0100

The other day, I came across an interesting thread in the Java sub-reddit, with someone asking: "Has anyone attempted to write logs directly to Kafka?". This triggered a number of thoughts and questions for myself, in particular how one should deal in an application when an attempt to send messages to Kafka fails, for instance due to some network connectivity issue? What do you do when you cannot reach the Kafka broker?

Refining The Return Type Of Java Methods Without Breaking Backwards-Compatibility

Mon, 22 Nov 2021 18:30:00 +0100

Table of Contents

If you work on any kind of software library, ensuring backwards-compatibility is a key concern: if there’s one thing which users really dislike, it is breaking changes in a new version of a library. The rules of what can (and cannot) be changed in a Java API without breaking existing consumers are well defined in the Java language specification (JLS), but things can get pretty interesting in certain corner cases.

The Eclipse team provides a comprehensive overview about API evolution guidelines in their wiki. When I shared the link to this great resource on Twitter the other day, I received an interesting reply from Lukas Eder:

I wish Java had a few tools to prevent some cases of binary compatibility breakages. E.g. when refining a method return type, I’d like to keep the old method around in byte code (but not in source code).

I think kotlin has such tools?

In the remainder of this post, I’d like to provide some more insight into that problem mentioned by Lukas, and how it can be addressed using an open-source tool called Bridger.

JDK Flight Recorder Events in GraalVM Native Binaries

Fri, 12 Nov 2021 18:30:00 +0100

If you have followed this blog for a while, you’ll know that I am a big fan of JDK Flight Recorder (JFR), the low-overhead diagnostics and profiling framework built into the HotSpot Java virtual machine. And indeed, until recently, this meant only HotSpot: Folks compiling their Java applications into GraalVM native binaries could not benefit from all the JFR goodness so far.

Debezium and Friends – Conference Talks 2021

Tue, 02 Nov 2021 10:50:00 +0100

Table of Contents Don’t Fear Outdated Caches – Change Data Capture to the Rescue! Change Data Streaming Patterns in Distributed Systems Analyzing Real-time Order Deliveries using CDC with Debezium and Pinot Dissecting our Legacy: The Strangler Fig Pattern with Apache Kafka, Debezium and MongoDB Bonus: Debezium at the Trino Community Broadcast Learning More If you love to attend conferences around the world without actually leaving the comfort of your house, 2021 certainly was (and is!

What's on My Desk?

Sun, 24 Oct 2021 10:30:00 +0200

Table of Contents

I’ve been working from home exclusively for the last nine years, but it was only last year that I started to look into ways for expanding my computer set-up and go beyond the usual combination of having a laptop with your regular external screen. The global COVID-19 pandemic, the prospect of having more calls with colleagues than ever (no physical meetings), and the constantly increasing need for recording talks for online conferences and meet-ups made me reevaluate things and steadily improve and fine tune my set-up, in particular in regards to better video and audio quality.

Executable JavaDoc Code Snippets

Mon, 18 Oct 2021 08:30:00 +0200

Table of Contents

It has been just a few weeks since the release of Java 17, but the first changes scheduled for Java 18 begin to show up in early access builds. One feature in particular that excites me as a maintainer of different Java libraries is JEP 413 ("Code Snippets in Java API Documentation").

Resource Bundle Look-ups in Modular Java Applications

Sun, 29 Aug 2021 18:30:00 +0200

Table of Contents

The ResourceBundle class is Java’s workhorse for managing and retrieving locale specific resources, such as error messages of internationalized applications. With the advent of the module system in Java 9, specifics around discovering and loading resource bundles have changed quite a bit, in particular when it comes to retrieving resource bundles across the boundaries of named modules.

In this blog post I’d like to discuss how resource bundles can be used in a multi-module application (i.e. a "modular monolith") for internationalizing error messages. The following requirements should be satisified:

Introducing JfrUnit 1.0.0.Alpha1

Wed, 04 Aug 2021 19:10:00 +0200

Table of Contents

Unit testing, for performance

It’s with great pleasure that I’m announcing the first official release of JfrUnit today!

JfrUnit is an extension to JUnit which allows you to assert JDK Flight Recorder events in your unit tests. This capability opens up a number of interesting use cases in the field of testing JVM-based applications:

You can use JfrUnit to ensure your application produces the custom JFR events you expect it to emit
You can use JfrUnit to identify potential performance regressions of your application by means of tracking JFR events e.g. for garbage collection, memory allocation and network I/O
You can use JfrUnit together with JMC Agent for whitebox tests of your application, ensuring specific methods are invoked with the expected parameters and return values

Three Plus Some Lovely Kafka Trends

Fri, 28 May 2021 10:30:00 +0200

Table of Contents

Over the course of the last few months, I’ve had the pleasure to serve on the Kafka Summit program committee and review several hundred session abstracts for the three Summits happening this year (Europe, APAC, Americas). That’s not only a big honour, but also a unique opportunity to learn what excites people currently in the Kafka eco-system (and yes, it’s a fair amount of work, too ;).

While voting on the proposals, and also generally aspiring to stay informed of what’s going on in the Kafka community at large, I noticed a few repeating themes and topics which I thought would be interesting to share (without touching on any specific talks of course). At first I meant to put this out via a Twitter thread, but then it became a bit too long for that, so I decided to write this quick blog post instead. Here it goes!

Exploring ZooKeeper-less Kafka

Mon, 17 May 2021 18:45:00 +0100

Table of Contents

Sometimes, less is more. One case where that’s certainly true is dependencies. And so it shouldn’t come at a surprise that the Apache Kafka community is eagerly awaiting the removal of the dependency to the ZooKeeper service, which currently is used for storing Kafka metadata (e.g. about topics and partitions) as well as for the purposes of leader election in the cluster.

The Kafka improvement proposal KIP-500 ("Replace ZooKeeper with a Self-Managed Metadata Quorum") promises to make life better for users in many regards:

Better getting started and operational experience by requiring to run only one system, Kafka, instead of two
Removing potential for discrepancies of metadata state between ZooKeeper and the Kafka controller
Simplifying configuration, for instance when it comes to security
Better scalability, e.g. in terms of number of partitions; faster execution of operations like topic creation

The Anatomy of ct.sym — How javac Ensures Backwards Compatibility

Mon, 26 Apr 2021 18:45:00 +0100

One of the ultimate strengths of Java is its strong notion of backwards compatibility: Java applications and libraries built many years ago oftentimes run without problems on current JVMs, and the compiler of current JDKs can produce byte code, that is executable with earlier Java versions.

For instance, JDK 16 supports byte code levels going back as far as to Java 1.7; But: hic sunt dracones. The emitted byte code level is just one part of the story. It’s equally important to consider which APIs of the JDK are used by the compiled code, and whether they are available in the targeted Java runtime version. As an example, let’s consider this simple "Hello World" program:

FizzBuzz – SIMD Style!

Mon, 08 Mar 2021 16:25:00 +0100

Table of Contents

Java 16 is around the corner, so there’s no better time than now for learning more about the features which the new version will bring. After exploring the support for Unix domain sockets a while ago, I’ve lately been really curious about the incubating Vector API, as defined by JEP 338, developed under the umbrella of Project Panama, which aims at "interconnecting JVM and native code".

Vectors?!? Of course this is not about renewing the ancient Java collection types like java.util.Vector (<insert some pun about this here>), but rather about an API which lets Java developers take advantage of the vector calculation capabilities you can find in most CPUs these days. Now I’m by no means an expert on low-level programming leveraging specific CPU instructions, but exactly that’s why I hope to make the case with this post that the new Vector API makes these capabilities approachable to a wide audience of Java programmers.

Talking to Postgres Through Java 16 Unix-Domain Socket Channels

Sun, 31 Jan 2021 20:25:00 +0100

Table of Contents

Reading a blog post about what’s coming up in JDK 16 recently, I learned that one of the new features is support for Unix domain sockets (JEP 380). Before Java 16, you’d have to resort to 3rd party libraries like jnr-unixsocket in order to use them. If you haven’t heard about Unix domain sockets before, they are "data communications [endpoints] for exchanging data between processes executing on the same host operating system". Don’t be put off by the name btw.; Unix domain sockets are also supported by macOS and even Windows since version 10.

jlink's Missing Link: API Signature Validation

Mon, 28 Dec 2020 12:20:00 +0100

Table of Contents

Discussions around Java’s jlink tool typically center around savings in terms of (disk) space. Instead of shipping an entire JDK, a custom runtime image created with jlink contains only those JDK modules which an application actually requires, resulting in smaller distributables and container images.

But the contribution of jlink — as a part of the Java module system at large — to the development of Java application’s is bigger than that: with the notion of link time it defines an optional complement to the well known phases compile time and application run-time:

Link time is an opportunity to do whole-world optimizations that are otherwise difficult at compile time or costly at run-time. An example would be to optimize a computation when all its inputs become constant (i.e., not unknown). A follow-up optimization would be to remove code that is no longer reachable.

ByteBuffer and the Dreaded NoSuchMethodError

Mon, 21 Dec 2020 17:45:00 +0100

Table of Contents

How to Prevent This Situation?

The other day, a user in the Debezium community reported an interesting issue; They were using Debezium with Java 1.8 and got an odd NoSuchMethodError:

Towards Continuous Performance Regression Testing

Wed, 16 Dec 2020 17:45:00 +0100

Table of Contents

Functional unit and integration tests are a standard tool of any software development organization, helping not only to ensure correctness of newly implemented code, but also to identify regressions — bugs in existing functionality introduced by a code change. The situation looks different though when it comes to regressions related to non-functional requirements, in particular performance-related ones: How to detect increased response times in a web application? How to identify decreased throughput?

These aspects are typically hard to test in an automated and reliable way in the development workflow, as they are dependent on the underlying hardware and the workload of an application. For instance assertions on the duration of specific requests of a web application typically cannot be run in a meaningful way on a developer laptop, which differs from the actual production hardware (ironically, nowadays both is an option, the developer laptop being less or more powerful than the actual production environment). When run in a virtualized or containerized CI environment, such tests are prone to severe measurement distortions due to concurrent load of other applications and jobs.

This post introduces the JfrUnit open-source project, which offers a fresh angle to this topic by supporting assertions not on metrics like latency/throughput themselves, but on indirect metrics which may impact those. JfrUnit allows you define expected values for metrics such as memory allocation, database I/O, or number of executed SQL statements, for a given workload and asserts the actual metrics values — which are obtained from JDK Flight Recorder events — against these expected values. Starting off from a defined base line, future failures of such assertions are an indicator for potential performance regressions in an application, as a code change may have introduced higher GC pressure, the retrieval of unneccessary data from the database, or SQL problems commonly induced by ORM tools, like N+1 SELECT statements.

Smaller, Faster-starting Container Images With jlink and AppCDS

Sun, 13 Dec 2020 20:30:00 +0100

Table of Contents

A few months ago I wrote about how you could speed up your Java application’s start-up times using application class data sharing (AppCDS), based on the example of a simple Quarkus application. Since then, quite some progress has been made in this area: Quarkus 1.6 brought built-in support for AppCDS, so that now you just need to provide the -Dquarkus.package.create-appcds=true option when building your project, and you’ll find an AppCDS file in the target folder.

Things get more challenging though when combining AppCDS with custom Java runtime images, as produced using the jlink tool added in Java 9. Combining custom runtime images with AppCDS is very attractive, in particular when looking at the deployment of Java applications via Linux containers. Instead of putting the full Java runtime into the container image, you only add those JDK modules which your application actually requires. (Parts of) what you save in image size by doing so, can be used for adding an AppCDS archive to your container image. The result will be a container image which still is smaller than before — and thus is faster to push to a container registry, distribute to worker nodes in a Kubernetes cluster, etc. — and which starts up significantly faster.

Quarkus and Testcontainers

Sat, 28 Nov 2020 17:45:00 +0100

Table of Contents

Bonus: Schema Creation

The Testcontainers project is invaluable for spinning up containerized resources during your (JUnit) tests, e.g. databases or Kafka clusters.

For users of JUnit 5, the project provides the @Testcontainers extension, which controls the lifecycle of containers used by a test. When testing a Quarkus application though, this is at odds with Quarkus' own @QuarkusTest extension; it’s a recommended best practice to avoid fixed ports for any containers started by Testcontainers. Instead, you should rely on Docker to automatically allocate random free ports. This avoids conflicts between concurrently running tests, e.g. amongst multiple Postgres containers, started up by several parallel job runs in a CI environment, all trying to allocate Postgres' default port 5432. Obtaining the randomly assigned port and passing it into the Quarkus bootstrap process isn’t possible though when combining the two JUnit extensions.

Class Unloading in Layered Java Applications

Wed, 14 Oct 2020 17:45:00 +0200

Table of Contents

Layers are sort of the secret sauce of the Java platform module system (JPMS): by providing fine-grained control over how individual JPMS modules and their classes are loaded by the JVM, they enable advanced usages like loading multiple versions of a given module, or dynamically adding and removing modules at application runtime.

The Layrry API and launcher provides a small plug-in API based on top of layers, which for instance can be used to dynamically add plug-ins contributing new views and widgets to a running JavaFX application. If such plug-in gets removed from the application again, all its classes need to be unloaded by the JVM, avoiding an ever-increasing memory consumption if for instance a plug-in gets updated multiple times.

In this blog post I’m going to explore how to ensure classes from removed plug-in layers are unloaded in a timely manner, and how to find the culprit in case some class fails to be unloaded.

Building hsdis for OpenJDK 15

Mon, 05 Oct 2020 20:30:00 +0200

Lately I’ve been fascinated by the possibility to analyse the assembly code emitted by the Java JIT (just-in-time) compiler. So far I had only looked only into Java class files using javap; diving into the world of assembly code feels a bit like Alice must have felt when falling down the rabbit whole into wonderland.

Introducing JmFrX: A Bridge From JMX to JDK Flight Recorder

Tue, 18 Aug 2020 17:30:00 +0200

Table of Contents

I’m excited to share the news about an open-source utility I’ve been working on lately: JmFrX, a tool for capturing JMX data with JDK Flight Recorder.

When using JMX (Java Management Extensions), The Java platform’s standard for monitoring and managing applications, JmFrX allows you to periodically record the attributes from any JMX MBean into JDK Flight Recorder (JFR) files, which you then can analyse using JDK Mission Control (JMC).

How I Built a Serverless Search for My Blog

Wed, 29 Jul 2020 17:30:00 +0200

Table of Contents

I have built a custom search functionality for this blog, based on Java and the Apache Lucene full-text search library, compiled into a native binary using the Quarkus framework and GraalVM. It is deployed as a Serverless application running on AWS Lambda, providing search results without any significant cold start delay. If you thought Java wouldn’t be the right language for this job, keep reading; in this post I’m going to give an overview over the implementation of this feature and my learnings along the way.

Building Class Data Sharing Archives with Apache Maven

Thu, 11 Jun 2020 17:30:00 +0200

Table of Contents

Ahead-of-time compilation (AOT) is the big topic in the Java ecosystem lately: by compiling Java code to native binaries, developers and users benefit from vastly improved start-up times and reduced memory usage. The GraalVM project made huge progress towards AOT-compiled Java applications, and Project Leyden promises to standardize AOT in a future version of the Java platform.

This makes it easy to miss out on significant performance improvements which have been made on the JVM in recent Java versions, in particular when it comes to faster start-up times. Besides a range of improvements related to class loading, linking and bytecode verification, substantial work has been done around class data sharing (CDS). Faster start-ups are beneficial in many ways: shorter turnaround times during development, quicker time-to-first-response for users in coldstart scenarios, cost savings when billed by CPU time in the cloud.

With CDS, class metadata is persisted in an archive file, which during subsequent application starts is mapped into memory. This is faster than loading the actual class files, resulting in reduced start-up times. When starting multiple JVM processes on the same host, read-only archives of class metadata can also be shared between the VMs, so that less memory is consumed overall.

Single Message Transformations - The Swiss Army Knife of Kafka Connect

Thu, 14 May 2020 15:30:00 +0200

Table of Contents

Do you remember Angus "Mac" MacGyver? The always creative protagonist of the popular 80ies/90ies TV show, who could solve about any problem with nothing more than a Swiss Army knife, duct tape, shoe strings and a paper clip?

The single message transformations (SMTs) of Kafka Connect are almost as versatile as MacGyver’s Swiss Army knife:

How to change the timezone or format of date/time message fields?
How to change the topic a specific message gets sent to?
How to filter out specific records?

SMTs can be the answer to these and many other questions that come up in the context of Kafka Connect. Applied to source or sink connectors, SMTs allow to modify Kafka records before they are sent to Kafka, or after they are consumed from a topic, respectively.

The Emitter Parameter Pattern for Flexible SPI Contracts

Mon, 04 May 2020 17:30:00 +0200

Table of Contents

For libraries and frameworks it’s a common requirement to make specific aspects customizeable via service provider interfaces (SPIs): contracts to be implemented by the application developer, which then are invoked by framework code, adding new or replacing existing functionality.

Often times, the method implementations of such an SPI need to return value(s) to the framework. An alternative to return values are "emitter parameters": passed by the framework to the SPI method, they offer an API for receiving value(s) via method calls. Certainly not revolutionary or even a new idea, I find myself using emitter parameters more and more in libraries and frameworks I work on. Hence I’d like to discuss some advantages I perceive about the emitter parameter pattern.

Plug-in Architectures With Layrry and the Java Module System

Tue, 21 Apr 2020 18:54:00 +0200

Table of Contents

Making applications extensible with some form of plug-ins is a very common pattern in software design: based on well-defined APIs provided by the application core, plug-ins can customize an application’s behavior and provide new functionality. Examples include desktop applications like IDEs or web browsers, build tools such as Apache Maven or Gradle, as well as server-side applications such as Apache Kafka Connect, a runtime for Kafka connectors plug-ins.

In this post I’m going to explore how the Java Platform Module System's notion of module layers can be leveraged for implementing plug-in architectures on the JVM. We’ll also discuss how Layrry, a launcher and runtime for layered Java applications, can help with this task.

Introducing Layrry: A Launcher and API for Modularized Java Applications

Sun, 29 Mar 2020 21:31:00 +0200

Table of Contents

One of the biggest changes in recent Java versions has been the introduction of the module system in Java 9. It allows to organize Java applications and their dependencies in strongly encapsulated modules, utilizing explicit and well-defined module APIs and relationships.

In this post I’m going to introduce the Layrry open-source project, a launcher and Java API for executing modularized Java applications. Layrry helps Java developers to assemble modularized applications from dependencies using their Maven coordinates and execute them using module layers. Layers go beyond the capabilities of the "flat" module path specified via the --module-path parameter of the java command, e.g. allowing to use multiple versions of one module within one and the same application.

Reworking Git Branches with git filter-branch

Mon, 16 Mar 2020 00:00:10 +0100

Table of Contents

Within Debezium, the project I’m working on at Red Hat, we recently encountered an "interesting" situation where we had to resolve a rather difficult merge conflict. As others where interested in how we addressed the issue, and also for our own future reference, I’m going to give a quick run down of the problem we encountered and how we solved it.

Monitoring REST APIs with Custom JDK Flight Recorder Events

Wed, 29 Jan 2020 15:30:00 +0100

Table of Contents

The JDK Flight Recorder (JFR) is an invaluable tool for gaining deep insights into the performance characteristics of Java applications. Open-sourced in JDK 11, JFR provides a low-overhead framework for collecting events from Java applications, the JVM and the operating system.

In this blog post we’re going to explore how custom, application-specific JFR events can be used to monitor a REST API, allowing to track request counts, identify long-running requests and more. We’ll also discuss how the JFR Event Streaming API new in Java 14 can be used to export live events, making them available for monitoring and alerting via tools such as Prometheus and Grafana.

Enforcing Java Record Invariants With Bean Validation

Mon, 20 Jan 2020 17:30:00 +0100

Table of Contents Record Invariants and Bean Validation Implementation Advantages Limitations Wrap-Up Record types are one of the most awaited features in Java 14; they promise to "provide a compact syntax for declaring classes which are transparent holders for shallowly immutable data". One example where records should be beneficial are data transfer objects (DTOs), as e.g. found in the remoting layer of enterprise applications. Typically, certain rules should be applied to the attributes of such DTO, e.

Using Java 13 Text Blocks (Only) for Your Tests

Mon, 13 Jan 2020 17:30:00 +0100

Table of Contents

When Java 9 was introduced in 2017, it was the last major version published under the old release scheme. Since then, a six month release cadence has been adopted. This means developers don’t have to wait years for new APIs and language features, but they can get their hands onto the latest additions twice a year. In this post I’d like to describe how you can try out new language features such as Java 13 text blocks in the test code of your project, while keeping your main code still compatible with older Java versions.

Quarkus Qute – A Test Ride

Fri, 03 Jan 2020 10:00:00 +0100

Table of Contents

One of the long-awaited features in Quarkus was support for server-side templating: until recently, Quarkus supported only client-side web frameworks which obtain there data by calling a REST API on the backend. This has changed with Quarkus 1.1: it comes with a brand-new template engine named Qute, which allows to build web applications using server-side templates.

Automatically Deploying a Hugo Website via GitHub Actions

Thu, 26 Dec 2019 10:00:00 +0100

Table of Contents

As a software engineer, I like to automate tedious tasks as much as possible. The deployment of this website is no exception: it is built using the Hugo static site generator and hosted on GitHub Pages; so wouldn’t it be nice if the rendered website would automatically be published whenever an update is pushed to its source code repository?

Time for a New Blog

Thu, 26 Dec 2019 10:00:00 +0100

It has been quite a while since the last post on my old personal blog; since then, I’ve mostly focused on writing about my day-work on the Debezium blog as well as some posts about more general technical topics on the Hibernate team blog.