It goes to python shell if it is successful. Along with this, pip (python package) also included. With this, we can install "pyspark" env in our system so that we can run spark related programs. But ...

When installing dagster-pyspark on EMR with pip, it forces the installation of pyspark 3.3.0. This is not correct, and breaks EMR. Launch EMR cluster with Spark Application enabled. SSH into master ...