I just got back from San Francisco after attending the Databricks Data + AI Summit 2025—and it’s hard to put into words how impactful the experience was. Over four packed days, I found myself reflecting not just on where the industry is going, but how the work we do as data engineers is evolving at a fundamental level.

This post isn’t a roundup of announcements. It’s a personal journal of sorts—what I learned, what made me rethink certain assumptions, and what I’m excited to bring back to my projects.

The Shift Toward Declarative Pipelines

One of the biggest takeaways for me was how Databricks is reimagining data engineering workflows through declarative pipelines. As someone who’s spent years designing DAGs, managing checkpoints, and writing stateful logic, it was refreshing (and a little mind-bending) to see that we’re now moving toward a model where we describe what the pipeline should do, not how it should do it.

This isn’t just a fancy abstraction. The declarative model supports:

• Auto-incremental processing (thanks to an “enzyme engine” that tracks changes),
• Unified support for batch and streaming,
• Built-in retries and DAG visualization,
• Git-based pipeline definitions.

To me, this represents more than a tooling shift—it’s a philosophical change. It gives back time to focus on data modeling and value delivery rather than wiring and orchestration. As someone who deeply values clean and maintainable pipelines, this resonated.

From DLT to Lakeflow: The Evolution

Delta Live Tables (DLT) was already powerful, but now it has matured into Lakeflow Declarative Pipelines. What’s changed? A lot.

There’s now a proper IDE for data engineering, complete with:

• Built-in CI/CD integration,
• Git-backed definitions,
• Reusable macros,
• Production-grade scheduling with triggers (e.g., file arrival, table changes).

Also, Lakeflow Jobs replaces Workflows with a more event-driven, robust orchestration engine. For large teams or regulated environments, this makes things less error-prone and more observable. Personally, I liked how clearly Lakeflow Jobs decouple compute from orchestration—especially when compared to the usual Airflow setups many teams (mine included) have juggled for years.

Spark 4.0 & the Open Lakehouse Standard

Spark 4.0 was another headline moment—and as someone who’s worked closely with Spark in production, I was particularly drawn to the new optimizer capabilities, enhanced Python APIs, and better support for semi-structured data.

But what stood out more was how Databricks is opening up declarative pipelines to the community. This isn’t vendor lock-in. It’s an intentional move to align with Apache Spark and extend the lakehouse to be truly open. It reinforces the idea that open standards are the future, and that the Databricks ecosystem is actively contributing back—not just innovating in isolation.

Zero-Bus Ingestion and Lakeflow Connect

Ingestion continues to be a major pain point in many pipelines. Databricks’ answer? Zero-Bus with Unity Catalog and Lakeflow Connect. What this means in practice is: you can ingest data from sources like PostgreSQL, Salesforce, and Workday with auto-CDC, fully governed metadata, and no need for custom buses or schedulers.

I tried a few demos and the whole process—setting up a connector, defining a table, configuring CDC—felt frictionless. As someone who’s had to write and maintain ingestion scripts across a half dozen tools, this consolidation is a welcome shift.

Real-Time Workloads That Don’t Feel Heavy

One of the most impressive parts of the Summit was seeing how real-time workloads are no longer treated as a special case. The same declarative pipeline can run in streaming or batch mode with no change in logic.

This was more than a demo. Teams showed real workloads where streaming jobs that used to take 10–15 minutes were now processing data within seconds. No magic—just incremental computation, checkpoint awareness, and optimized scheduling under the hood.

The big shift here for me was: you don’t need to make trade-offs anymore between latency and simplicity. As someone who’s usually cautious about streaming because of its operational overhead, this changes the game.

Learnings from the Trainings

I took a set of advanced, hands-on trainings while at the summit. Here’s what stood out:

• Databricks Performance Optimization: Learned how small things—like Z-ordering, liquid clustering, cluster tuning, and adaptive joins—can make a massive difference in cost and latency.
• Streaming and Declarative Pipelines: Gained practical experience with the new Lakeflow model. The enzyme engine’s ability to only process new data blew me away. It’s like a smart cache for your logic.
• Lakeflow Connect: Built ingestion pipelines using CDC connectors with minimal setup. I appreciated how quickly I could go from source to live Lakehouse tables.
• Governance with Unity Catalog: This one made me rethink access control. I now have better clarity on how lineage, policies, and audit logs come together to create a secure and compliant ecosystem.
• GenAI Application Development: Got hands-on with Mosaic and Agent Bricks. There’s a growing maturity in how these agents are being productized. I think we’ll see more hybrid roles emerge—data + AI developer.

Keynote Highlights

According to me, this year’s sessions were more grounded in practical insight than abstract vision. The following keynotes were big highlights or me:

• Ali Ghodsi outlined a clear north star: “data intelligence for all.” It’s not just AI hype—it’s about reducing the time from idea to production for everyone.
• Satya Nadella (Microsoft) brought up the idea of digital coworkers. I like that framing. AI isn’t a threat; it’s a tool that helps us focus on the deeper parts of our jobs.
• Dario Amodei (Anthropic CEO) shared an important reminder: responsibility in AI starts with how we shape and constrain its decision space.

There was also a great session by Michael Armbrust, who walked through the entire Spark 4.0 stack and how Lakeflow is built to scale with minimal friction.

Final Thoughts

What struck me most was the clarity. The future of data engineering isn’t about stitching together 10 tools. It’s about building declarative, governed, and scalable workflows—fast. And as someone who’s often at the intersection of engineering and advocacy, I see a huge opportunity here. This isn’t just a toolkit shift—it’s a mindset shift. If you’re in data engineering, now’s a good time to revisit your assumptions.

Let’s talk if you want to dig into any of these topics or see how they might apply to what we’re doing.

(Image credits go to their original creators and copyright owners)