Java Fails to Run Large Python Files

I have a Python script which I am attempting to run via code in Java.

The Python script runs fine when run through a Linux terminal command on my Ubuntu virtual machine using an identical command to the one being passed through the Java script.

The Java code runs fine when running a different Python script that runs faster than the Python script I'm attempting to run..

However, despite both the Python script running fine and the Java script running fine, somehow, when I put the two together, nothing happens: The .txt file isn't updated, so the Java script prints out whatever old value it contains.

System.out.println("starting...");
try {
    Process process = Runtime.getRuntime().exec("python3 /home/.../PycharmProjects/.../fraudanalysis.py abc def");
    Thread.sleep(900000);
    # Or try System.out.println(process.waitFor());
    File file = new File("/home/.../PycharmProjects/.../output.txt");
    Scanner newLineReader = new Scanner(file);
    System.out.println(newLineReader.nextLine());
} catch(Exception e) {
    System.out.println(e);
}

The code above should run the Python3 script at the absolute directory provided, using two arguments. The Python3 script completes after around 13 minutes and updates the output.txt file, which is then read by the Java program after waiting 15 minutes (or you can tell the thread to wait for completion-- process.WaitFor() returns 1).

def testScript():
    time.sleep(780)
    return_string1 = sys.argv[1]
    return_string2 = sys.argv[2]

    outputFile = open(os.path.dirname(os.path.abspath(__file__)) + "/output/output.txt", "w+")
    outputFile.write(return_string1 + " " + return_string2)
    print("Python run complete")

if __name__ == "__main__":
    testScript()

The script above is a good stand-in for the Python script. If you lower the sleep time to 10 minutes for the Python script, it runs when Java sends the command. But, at the sleep times shown above, Java apparently fails to run the script, or the script run attempt ends in failure.

Additional info: the Java command is activated using a JavaFX button. The Java script has been developed in IntelliJ IDEA and the Python script was created using PyCharm.

My question is, what are possible causes for this problem, when both scripts work fine on their own?

As a simple suggestion, you should not rely on Thread.sleep method with a fixed parameter such as 15 minutes. Your data may grow or shrink and that way of proceeding is not efficient.

You could try to call the Process.waitFor() method so that when the python process is over, your thread continues.

Moreover, you could try to use ProcessBuilder that sometimes helps when facing buggy System exec cases.

Here is some code. in sub(), you can not change the python program, but for sub2() to work, you have to modify the python program so that its output is on the standard out and Java would do the redirect to the output.txt file.

public void sub() {
System.out.println("startig...");
Scanner newLineReader = null;
try {
Process process = Runtime.getRuntime().exec("python3 /home/.../PycharmProjects/.../fraudanalysis.py /home/.../PycharmProjects/.../fraudAnalysis.db 500");
process.waitFor();
File file = new File("/home/.../PycharmProjects/.../output.txt");
newLineReader = new Scanner(file);
String line;
while((line=newLineReader.nextLine())!=null) {
System.out.println(line);
}
} catch(IOException ioe) {
ioe.printStackTrace();
}catch(InterruptedException ie) {
ie.printStackTrace();
}finally {
newLineReader.close();
}
}
public void sub2() {
ProcessBuilder pb =
new ProcessBuilder("python3", 
"/home/.../PycharmProjects/.../fraudanalysis.py", 
"/home/.../PycharmProjects/.../fraudAnalysis.db", "500");
File log = new File("/home/.../PycharmProjects/.../output.txt");
pb.redirectErrorStream(true);
pb.redirectOutput(Redirect.appendTo(log));
Process p = null;
try {
p = pb.start();
} catch (IOException e) {
e.printStackTrace();
}
try {
p.waitFor();
} catch (InterruptedException e) {
e.printStackTrace();
}
Scanner newLineReader = null;
try{
newLineReader = new Scanner(log);
String line;
while((line=newLineReader.nextLine())!=null) {
System.out.println(line);
}
}catch(IOException ioe) {
ioe.printStackTrace();
}
}

I was able to get it to work with a small modification. I used relative file locations and TimeUnit.MINUTES.sleep(15);

package org.openjfx;
import java.io.File;
import java.util.Scanner;
import java.util.concurrent.TimeUnit;
public class TestWait {
public static void main(String[] args) {
System.out.println("starting...");
String dir="src/main/resources/org/openjfx/";//location of the python script
try {
System.out.println("Working Directory = " + System.getProperty("user.dir"));
//System.out.println("python3 " + dir+"fraudanalysis.py abc def");
Process process = Runtime.getRuntime().exec("python3 " + dir+"fraudanalysis.py abc def");
System.out.println(process.waitFor());
TimeUnit.MINUTES.sleep(15);
File file = new File("src/main/resources/org/openjfx/output.txt");
Scanner newLineReader = new Scanner(file);
System.out.println(newLineReader.nextLine());
} catch (Exception e) {
System.out.println(e);
}
}
}

Here is the python I used.

import sys
import time
def testScript():
return_string1 = sys.argv[1]
return_string2 = sys.argv[2]
time.sleep(780)
outputFile = open("src/main/resources/org/openjfx/output.txt", "w+")
outputFile.write(return_string1 + " " + return_string2)
print("Python run complete")
if __name__ == "__main__":
testScript()

it's a timeout error. can't be fixed. just pick between Java and Python and write everything in it. no reason to use both.