Provides a 'PySpark' Back-End for the 'sparklyr' Package

It enables 'sparklyr' to integrate with 'Spark Connect', and 'Databricks Connect' by providing a wrapper over the 'PySpark' 'python' library.

pysparklyr

Integrates sparklyr with PySpark and Databricks. The main reason of this package is because the new Spark and Databricks Connect connection method does not work with standard sparklyr integration.

Installing

To install the version in CRAN use:

install.packages("pysparklyr")

To get the development version from GitHub use:

remotes::install_github("mlverse/pysparklyr")

Using

To learn how to use, please visit the Spark / Databricks Connect article, available in the official sparklyr website: Spark Connect, and Databricks Connect v2