r/scala

We have open sourced Wick, A zero cost type safe Apache Spark API!

I wrote this blog post to scratch an itch; I wanted to learn about multi-stage programming and implemented a toy parser combinator library where I explore several common patterns, and do a final benchmark to see whether it's worth the added complexity.

I'm in the middle of a migration that involves rewriting a lot of .proto files, and I needed a way to check which changes would break generated client code vs. only break on-the-wire compatibility. buf breaking is wire-first, but during a migration where every client is rebuilt from the new schema, wire compat isn't what I cared about.

Proteus-diff treats wire and source compatibility as independent axes. Same diff, different answer depending on which one matters to you. Example of two fields swap numbers:

User (2)
  error [FieldNumberChanged] field 'name' number changed from 1 to 2
  error [FieldNumberChanged] field 'email' number changed from 2 to 1

Under --mode wire that's two errors (old bytes decode into the wrong field). Under --mode source it's two infos (generated code doesn't care about numbers).

The tool is shipped as part of the open-source Proteus Scala library, but the CLI is standalone and works on any `.proto` files. It is built with Fastparse and Mainargs, packaged with GraalVM native image and only takes 250ms to parse and compare ~100 files with ~6,000 messages. Hope it can be useful to others!

apply and unapply + pattern matching is the strong case for case classes. I created this article on my webpage https://codeinvestigator.com/articles/scala-is-more-fun-than-java

And then the boss fight appears:

“Which Java version should we choose?”

If your build matrix looks like a sci-fi timeline and IntelliJ has opinions, this one’s for you.

sbt 2 project has started the last mile process to lock down the 2.0.x branch. Depending on the bugs we discover, we are hopeful that a release candidate (currently 2.0.0-RC12) can graduate in a few weeks to a few months.

We have a few Spark jobs that are very similar in terms of logic and structure. They run on the same cluster with the same configs

In theory performance should be close, but in practice it isn’t. Some runs finish in around 10–12 minutes, others go past 20 minutes with no clear difference in input size

Checked Spark UI, executors, stages, shuffle behavior. Nothing stands out. No failures, no obvious skew

This started showing up more once more jobs were added to the cluster. Feels like resource contention but not fully clear where it shows up

Has anyone seen this kind of variation across similar Spark jobs and what usually causes it

When working with multiple Scala projects a day I sometimes steuggle with large resource consumption, like leftover sbt sessions or metals instances. This is why I created Scala monitor - simple utility to list all scala related processes along with the project the process is related to. Nothing sophisticated, just glorified ps/grep, but solves my problem, and makes a great use case for Scala Native.

Check it out https://github.com/polyvariant/scala-monitor

I am at a bit of a crossroads in my career right now. I have around 7 years of dev experience in Scala and 4 of those are in SWE and I have 3 years as a Data Engineer. I changed my job because it was more money and unwanted to do something different. Now, I am kind of bored of this surface level scala and spark programming and want to get back into SWE.

I find that functional programming with cats and cats effect kind of itches that spot in my brain if you know what I mean. My question is will recruiters see me as a potential hire as someone whose past 3 years is only as a Data Engineer?

I have worked with the effects system before but that was for about a year and I feel like the SWE world has changed a lot since I moved on a while back.

I am currently doing a lot of self study and personal projects to familiarise myself with the ecosystem again especially CE3.

If anyone could advise me on what I could do to make myself more marketable or if recruiters will even see me as a potential hire, that would be really appreciated.

Scala devs, assemble: Lists, Sets, and Maps walk into a codebase… only one allows duplicates, one judges them, and one turns everything into key-value drama.

If your collections knowledge feels a bit mutable, this quick video helps sort it out:

Scala Lists and Sets and Maps

Hi everyone, I have written a cheat sheet containing over 50+ http clients configured with SSL and also with an example request. It contains next to Scala also clients for other jvm languages such as Java, Kotlin, Clojure, and Groovy. Feel free to share your thoughts