It is well known — or should be — that spark is not secured by default. It is right there in the docs Security in Spark is OFF by default So you should be well aware that you’ll need to put the effort to secure your cluster. And there are many th...
Interpreting Isolation Forest’s predictions — and not only Photo by Joshua Golde on Unsplash The problem: how to interpret Isolation Forest’s predictions More specifically, how to tell which features are contributing more to the predictions. Since Is...
Time to check your spark clusters Image by Author It is well known — or should be — that spark is not secured by default. It is right there in the docs Security in Spark is OFF by default So you should be well aware that you’ll need to put the effo...
Lessons learned Debugging PySpark and Isolation Forest — Image by author So, after a few runs with the PySpark ml implementation of Isolation Forest presented here, I stumbled upon a couple of things and I thought I’d write about them so that you don...
How to leverage spark to read in parallel from a database Spark Parallelization A usual way to read from a database, e.g. Postgres, using spark would be something like the following: user = 'postgres' password = 'secret' db_driver = 'org.postgresql.D...
Superpower I wish I had : Telekinesis Photo by Lacie Slezak on Unsplash As a first time mom of a now two-month old beautiful baby, I find myself most of the time confined in awkward and tiring positions to feed her and to help her relax and sleep. My...
Main characteristics and ways to use Isolation Forest in PySpark Isolation Forest is an algorithm for anomaly / outlier detection, basically a way to spot the odd one out. We go through the main characteristics and explore two ways to use Isolation...
Photo by Ozgu Ozden on Unsplash For those who are familiar with pandas DataFrames, switching to PySpark can be quite confusing. The API is not the same, and when switching to a distributed nature, some things are being done quite differently because ...
Debugging a spark application can range from a fun to a very (and I mean very) frustrating experience. I’ve started gathering the issues I’ve come across from time to time to compile a list of the most common problems and their solutions. This is the...
