英文:
Is SSH remote debugging available for IntelliJ community edition ? If not, are there any alternatives?
问题
我需要在远程服务器上运行一个Spark应用程序，通过SSH可以访问该服务器。目前，我的团队在本地开发，使用Maven编译项目，通过SFTP将项目发送到服务器，然后从终端运行它并阅读日志以查看是否出现问题。我想知道是否可以将IntelliJ与远程服务器连接，并以调试模式运行Spark，以便我可以从IntelliJ远程调试，设置断点等。这是否可能？我是IntelliJ的新手，不确定这个远程功能是否仅在ultimate版中可用。如果是这样，还有什么其他解决方案吗？谢谢！
英文:
I need to work on a Spark application running on a remote server, on which I have access through SSH. At the moment, my team develop locally, compile the project with Maven, send it with SFTP to the server, run it from the terminal and read the logs to see if something went wrong. I would like to know if it is possible to link IntelliJ with the remote server and run Spark in debug mode so that I can debug remotely from IntelliJ, set breakpoints, etc. Is it possible ? I'm new to IntelliJ and I'm not sure to understand if this remote feature is only available in the ultimate edition. If it is the case, what other solution is there ? Thanks !
答案1
得分: 1
可以将IntelliJ的远程调试器连接到远程的Spark应用程序。实际上，您可以将调试器连接到正在执行您的应用程序的特定执行器上。您需要按照@y.bedrov在https://medium.com/agile-lab-engineering/spark-remote-debugging-371a1a8c44a8中指定的步骤进行操作。在您的情况下，我认为调试器无法连接，因为您将localhost指定为主机。您需要将其替换为应用程序正在执行的主机，换句话说，执行器主机。
英文:
It is possible to connect IntelliJ remote debugger to your remote Spark application.
Actually, you can connect the debugger to a specific executor on which your application is being executed.
You need to follow the steps as @y.bedrov specified https://medium.com/agile-lab-engineering/spark-remote-debugging-371a1a8c44a8
In your case, i think the debugger is not connecting because you are specifying localhost as the host.
You need to replace it with the host on which the application is being executed, in other words the executor host.
答案2
得分: 0
It's possible for general-purpose java applications such as spring boot web apps. You could connect to an existing running process with idea and debug it the same way, as you debug your local application (the process on the remote server should be executed with a special key that allowed remote debugging).
But when you run the spark application in a cluster there is no particular process to connect to, because the resource manager selects one node as a driver and some other nodes as executors, start the application on the driver, and performs distributed operations on executors. So you actually don't know the address of the host, where your application will be executed.
Therefore you can't connect to it and debug it.
So the answer is no - you can't use remote debug with spark application at all.
But, you could start the spark session locally on your own PC and debug it.
英文:
It's possible for general-purpose java applications such as spring boot web apps. You could connect to an existing running process with idea and debug it the same way, as you debug your local application (the process on the remote server should be executed with a special key that allowed remote debugging).
But when you run the spark application in a cluster there is no particular process to connect to, because the resource manager selects one node as a driver and some other nodes as executors, start the application on the driver, and performs distributed operations on executors. So you actually don't know the address of the host, where your application will be executed.
Therefore you can't connect to it and debug it.
So the answer is no - you can't use remote debug with spark application at all.
But, you could start the spark session locally on your own PC and debug it.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论