Dynamic output from python subprocess module
up vote
1
down vote
favorite
how can i achieve output dynamically using subprocess module (while the external program keeps running) in python. The external program from which i want to get output dynamically is ngrok ,
ngrok keep running as long as my program is running but i need output while the process is running so that i can extract the newly generated "forwarding url"
when i try to do :
cmd = ['ngrok', 'http', '5000']
output = subprocess.Popen(cmd, stdout=subprocess.PIPE, buffersize=1)
it keep storing output in buffers
python python-3.x subprocess ngrok
add a comment |
up vote
1
down vote
favorite
how can i achieve output dynamically using subprocess module (while the external program keeps running) in python. The external program from which i want to get output dynamically is ngrok ,
ngrok keep running as long as my program is running but i need output while the process is running so that i can extract the newly generated "forwarding url"
when i try to do :
cmd = ['ngrok', 'http', '5000']
output = subprocess.Popen(cmd, stdout=subprocess.PIPE, buffersize=1)
it keep storing output in buffers
python python-3.x subprocess ngrok
2
Possible duplicate of Printing output in realtime from subprocess. Altho that's just on the question itself, but it won't work for ngrok or other ncurses application. So just leaving this here for others who end up here wondering how to get output fromsubprocess
.
– Torxed
20 hours ago
add a comment |
up vote
1
down vote
favorite
up vote
1
down vote
favorite
how can i achieve output dynamically using subprocess module (while the external program keeps running) in python. The external program from which i want to get output dynamically is ngrok ,
ngrok keep running as long as my program is running but i need output while the process is running so that i can extract the newly generated "forwarding url"
when i try to do :
cmd = ['ngrok', 'http', '5000']
output = subprocess.Popen(cmd, stdout=subprocess.PIPE, buffersize=1)
it keep storing output in buffers
python python-3.x subprocess ngrok
how can i achieve output dynamically using subprocess module (while the external program keeps running) in python. The external program from which i want to get output dynamically is ngrok ,
ngrok keep running as long as my program is running but i need output while the process is running so that i can extract the newly generated "forwarding url"
when i try to do :
cmd = ['ngrok', 'http', '5000']
output = subprocess.Popen(cmd, stdout=subprocess.PIPE, buffersize=1)
it keep storing output in buffers
python python-3.x subprocess ngrok
python python-3.x subprocess ngrok
asked yesterday
arslanmughal
63
63
2
Possible duplicate of Printing output in realtime from subprocess. Altho that's just on the question itself, but it won't work for ngrok or other ncurses application. So just leaving this here for others who end up here wondering how to get output fromsubprocess
.
– Torxed
20 hours ago
add a comment |
2
Possible duplicate of Printing output in realtime from subprocess. Altho that's just on the question itself, but it won't work for ngrok or other ncurses application. So just leaving this here for others who end up here wondering how to get output fromsubprocess
.
– Torxed
20 hours ago
2
2
Possible duplicate of Printing output in realtime from subprocess. Altho that's just on the question itself, but it won't work for ngrok or other ncurses application. So just leaving this here for others who end up here wondering how to get output from
subprocess
.– Torxed
20 hours ago
Possible duplicate of Printing output in realtime from subprocess. Altho that's just on the question itself, but it won't work for ngrok or other ncurses application. So just leaving this here for others who end up here wondering how to get output from
subprocess
.– Torxed
20 hours ago
add a comment |
1 Answer
1
active
oldest
votes
up vote
0
down vote
I know this is a duplicate, but I can't find any relevant threads about this now. All i get is output.communicate()
.
So here's a snippet that might be useful:
import subprocess
cmd = ['ngrok', 'http', '5000']
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while process.poll() is None:
print(process.stdout.readline())
print(process.stdout.read())
process.stdout.close()
This would output anything the process outputs, through your script into your output. It does so by looking for a newline character before outputting.
This piece of code would work, if it weren't for the fact that ngrok
uses ncurses and/or hogs the output to it's own user/thread much like when SSH asks for a password when you do ssh user@host
.
process.poll()
checks if the process has an exit-code (if it's dead), if not, it continues to loop and print anything from the process's stdout
.
There's other (better) ways to go about it, but this is the bare minimum I can give you without it being complicated really fast.
For instance, process.stdout.read()
could be used in junction with select.select()
to achieve better buffered output where new-lines are scares. Because if a n
never comes, the above example might hang your entire application.
There's a lot of buffer-traps here that you need to be aware of before you do manual things like this. Otherwise, use process.communicate()
instead.
Edit: To get around the hogging/limitation of I/O used by ngrok, you could use pty.fork()
and read the childs stdout via the os.read
module:
#!/usr/bin/python
## Requires: Linux
## Does not require: Pexpect
import pty, os
from os import fork, waitpid, execv, read, write, kill
def pid_exists(pid):
"""Check whether pid exists in the current process table."""
if pid < 0:
return False
try:
kill(pid, 0)
except (OSError, e):
return e.errno == errno.EPERMRM
else:
return True
class exec():
def __init__(self):
self.run()
def run(self):
command = [
'/usr/bin/ngrok',
'http',
'5000'
]
# PID = 0 for child, and the PID of the child for the parent
pid, child_fd = pty.fork()
if not pid: # Child process
# Replace child process with our SSH process
execv(command[0], command)
while True:
output = read(child_fd, 1024)
print(output.decode('UTF-8'))
lower = output.lower()
# example input (if needed)
if b'password:' in lower:
write(child_fd, b'some responsen')
waitpid(pid, 0)
exec()
There's still a problem here, and I'm not quite sure what or why that is.
I'm guessing the process is waiting for a signal/flush some how.
The problem is that it's only printing the first "setup data" of ncurses, meaning it wips the screen and sets up the background-color.
But this would give you the output of the process at least. replacing print(output.decode('UTF-8'))
would show you what that output is.
it looks that program stuck at this line of code """process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)""" after reaching this line the program start storing output in buffers and become silent .. the only thing i want to extract from ngrok while ngrok is running is the farwarded url that ngrok provides
– arslanmughal
yesterday
@arslanmughal You are partially correct, it's not stuck on thePopen()
line tho. It's stuck onprint(process.stdout.readline())
. This is a huge different because it just tells us thatPopen()
isn't able to get the output from the process. I also triedstdout.read(1)
which returns nothing. So I added an additional way of getting the output.
– Torxed
20 hours ago
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
0
down vote
I know this is a duplicate, but I can't find any relevant threads about this now. All i get is output.communicate()
.
So here's a snippet that might be useful:
import subprocess
cmd = ['ngrok', 'http', '5000']
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while process.poll() is None:
print(process.stdout.readline())
print(process.stdout.read())
process.stdout.close()
This would output anything the process outputs, through your script into your output. It does so by looking for a newline character before outputting.
This piece of code would work, if it weren't for the fact that ngrok
uses ncurses and/or hogs the output to it's own user/thread much like when SSH asks for a password when you do ssh user@host
.
process.poll()
checks if the process has an exit-code (if it's dead), if not, it continues to loop and print anything from the process's stdout
.
There's other (better) ways to go about it, but this is the bare minimum I can give you without it being complicated really fast.
For instance, process.stdout.read()
could be used in junction with select.select()
to achieve better buffered output where new-lines are scares. Because if a n
never comes, the above example might hang your entire application.
There's a lot of buffer-traps here that you need to be aware of before you do manual things like this. Otherwise, use process.communicate()
instead.
Edit: To get around the hogging/limitation of I/O used by ngrok, you could use pty.fork()
and read the childs stdout via the os.read
module:
#!/usr/bin/python
## Requires: Linux
## Does not require: Pexpect
import pty, os
from os import fork, waitpid, execv, read, write, kill
def pid_exists(pid):
"""Check whether pid exists in the current process table."""
if pid < 0:
return False
try:
kill(pid, 0)
except (OSError, e):
return e.errno == errno.EPERMRM
else:
return True
class exec():
def __init__(self):
self.run()
def run(self):
command = [
'/usr/bin/ngrok',
'http',
'5000'
]
# PID = 0 for child, and the PID of the child for the parent
pid, child_fd = pty.fork()
if not pid: # Child process
# Replace child process with our SSH process
execv(command[0], command)
while True:
output = read(child_fd, 1024)
print(output.decode('UTF-8'))
lower = output.lower()
# example input (if needed)
if b'password:' in lower:
write(child_fd, b'some responsen')
waitpid(pid, 0)
exec()
There's still a problem here, and I'm not quite sure what or why that is.
I'm guessing the process is waiting for a signal/flush some how.
The problem is that it's only printing the first "setup data" of ncurses, meaning it wips the screen and sets up the background-color.
But this would give you the output of the process at least. replacing print(output.decode('UTF-8'))
would show you what that output is.
it looks that program stuck at this line of code """process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)""" after reaching this line the program start storing output in buffers and become silent .. the only thing i want to extract from ngrok while ngrok is running is the farwarded url that ngrok provides
– arslanmughal
yesterday
@arslanmughal You are partially correct, it's not stuck on thePopen()
line tho. It's stuck onprint(process.stdout.readline())
. This is a huge different because it just tells us thatPopen()
isn't able to get the output from the process. I also triedstdout.read(1)
which returns nothing. So I added an additional way of getting the output.
– Torxed
20 hours ago
add a comment |
up vote
0
down vote
I know this is a duplicate, but I can't find any relevant threads about this now. All i get is output.communicate()
.
So here's a snippet that might be useful:
import subprocess
cmd = ['ngrok', 'http', '5000']
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while process.poll() is None:
print(process.stdout.readline())
print(process.stdout.read())
process.stdout.close()
This would output anything the process outputs, through your script into your output. It does so by looking for a newline character before outputting.
This piece of code would work, if it weren't for the fact that ngrok
uses ncurses and/or hogs the output to it's own user/thread much like when SSH asks for a password when you do ssh user@host
.
process.poll()
checks if the process has an exit-code (if it's dead), if not, it continues to loop and print anything from the process's stdout
.
There's other (better) ways to go about it, but this is the bare minimum I can give you without it being complicated really fast.
For instance, process.stdout.read()
could be used in junction with select.select()
to achieve better buffered output where new-lines are scares. Because if a n
never comes, the above example might hang your entire application.
There's a lot of buffer-traps here that you need to be aware of before you do manual things like this. Otherwise, use process.communicate()
instead.
Edit: To get around the hogging/limitation of I/O used by ngrok, you could use pty.fork()
and read the childs stdout via the os.read
module:
#!/usr/bin/python
## Requires: Linux
## Does not require: Pexpect
import pty, os
from os import fork, waitpid, execv, read, write, kill
def pid_exists(pid):
"""Check whether pid exists in the current process table."""
if pid < 0:
return False
try:
kill(pid, 0)
except (OSError, e):
return e.errno == errno.EPERMRM
else:
return True
class exec():
def __init__(self):
self.run()
def run(self):
command = [
'/usr/bin/ngrok',
'http',
'5000'
]
# PID = 0 for child, and the PID of the child for the parent
pid, child_fd = pty.fork()
if not pid: # Child process
# Replace child process with our SSH process
execv(command[0], command)
while True:
output = read(child_fd, 1024)
print(output.decode('UTF-8'))
lower = output.lower()
# example input (if needed)
if b'password:' in lower:
write(child_fd, b'some responsen')
waitpid(pid, 0)
exec()
There's still a problem here, and I'm not quite sure what or why that is.
I'm guessing the process is waiting for a signal/flush some how.
The problem is that it's only printing the first "setup data" of ncurses, meaning it wips the screen and sets up the background-color.
But this would give you the output of the process at least. replacing print(output.decode('UTF-8'))
would show you what that output is.
it looks that program stuck at this line of code """process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)""" after reaching this line the program start storing output in buffers and become silent .. the only thing i want to extract from ngrok while ngrok is running is the farwarded url that ngrok provides
– arslanmughal
yesterday
@arslanmughal You are partially correct, it's not stuck on thePopen()
line tho. It's stuck onprint(process.stdout.readline())
. This is a huge different because it just tells us thatPopen()
isn't able to get the output from the process. I also triedstdout.read(1)
which returns nothing. So I added an additional way of getting the output.
– Torxed
20 hours ago
add a comment |
up vote
0
down vote
up vote
0
down vote
I know this is a duplicate, but I can't find any relevant threads about this now. All i get is output.communicate()
.
So here's a snippet that might be useful:
import subprocess
cmd = ['ngrok', 'http', '5000']
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while process.poll() is None:
print(process.stdout.readline())
print(process.stdout.read())
process.stdout.close()
This would output anything the process outputs, through your script into your output. It does so by looking for a newline character before outputting.
This piece of code would work, if it weren't for the fact that ngrok
uses ncurses and/or hogs the output to it's own user/thread much like when SSH asks for a password when you do ssh user@host
.
process.poll()
checks if the process has an exit-code (if it's dead), if not, it continues to loop and print anything from the process's stdout
.
There's other (better) ways to go about it, but this is the bare minimum I can give you without it being complicated really fast.
For instance, process.stdout.read()
could be used in junction with select.select()
to achieve better buffered output where new-lines are scares. Because if a n
never comes, the above example might hang your entire application.
There's a lot of buffer-traps here that you need to be aware of before you do manual things like this. Otherwise, use process.communicate()
instead.
Edit: To get around the hogging/limitation of I/O used by ngrok, you could use pty.fork()
and read the childs stdout via the os.read
module:
#!/usr/bin/python
## Requires: Linux
## Does not require: Pexpect
import pty, os
from os import fork, waitpid, execv, read, write, kill
def pid_exists(pid):
"""Check whether pid exists in the current process table."""
if pid < 0:
return False
try:
kill(pid, 0)
except (OSError, e):
return e.errno == errno.EPERMRM
else:
return True
class exec():
def __init__(self):
self.run()
def run(self):
command = [
'/usr/bin/ngrok',
'http',
'5000'
]
# PID = 0 for child, and the PID of the child for the parent
pid, child_fd = pty.fork()
if not pid: # Child process
# Replace child process with our SSH process
execv(command[0], command)
while True:
output = read(child_fd, 1024)
print(output.decode('UTF-8'))
lower = output.lower()
# example input (if needed)
if b'password:' in lower:
write(child_fd, b'some responsen')
waitpid(pid, 0)
exec()
There's still a problem here, and I'm not quite sure what or why that is.
I'm guessing the process is waiting for a signal/flush some how.
The problem is that it's only printing the first "setup data" of ncurses, meaning it wips the screen and sets up the background-color.
But this would give you the output of the process at least. replacing print(output.decode('UTF-8'))
would show you what that output is.
I know this is a duplicate, but I can't find any relevant threads about this now. All i get is output.communicate()
.
So here's a snippet that might be useful:
import subprocess
cmd = ['ngrok', 'http', '5000']
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while process.poll() is None:
print(process.stdout.readline())
print(process.stdout.read())
process.stdout.close()
This would output anything the process outputs, through your script into your output. It does so by looking for a newline character before outputting.
This piece of code would work, if it weren't for the fact that ngrok
uses ncurses and/or hogs the output to it's own user/thread much like when SSH asks for a password when you do ssh user@host
.
process.poll()
checks if the process has an exit-code (if it's dead), if not, it continues to loop and print anything from the process's stdout
.
There's other (better) ways to go about it, but this is the bare minimum I can give you without it being complicated really fast.
For instance, process.stdout.read()
could be used in junction with select.select()
to achieve better buffered output where new-lines are scares. Because if a n
never comes, the above example might hang your entire application.
There's a lot of buffer-traps here that you need to be aware of before you do manual things like this. Otherwise, use process.communicate()
instead.
Edit: To get around the hogging/limitation of I/O used by ngrok, you could use pty.fork()
and read the childs stdout via the os.read
module:
#!/usr/bin/python
## Requires: Linux
## Does not require: Pexpect
import pty, os
from os import fork, waitpid, execv, read, write, kill
def pid_exists(pid):
"""Check whether pid exists in the current process table."""
if pid < 0:
return False
try:
kill(pid, 0)
except (OSError, e):
return e.errno == errno.EPERMRM
else:
return True
class exec():
def __init__(self):
self.run()
def run(self):
command = [
'/usr/bin/ngrok',
'http',
'5000'
]
# PID = 0 for child, and the PID of the child for the parent
pid, child_fd = pty.fork()
if not pid: # Child process
# Replace child process with our SSH process
execv(command[0], command)
while True:
output = read(child_fd, 1024)
print(output.decode('UTF-8'))
lower = output.lower()
# example input (if needed)
if b'password:' in lower:
write(child_fd, b'some responsen')
waitpid(pid, 0)
exec()
There's still a problem here, and I'm not quite sure what or why that is.
I'm guessing the process is waiting for a signal/flush some how.
The problem is that it's only printing the first "setup data" of ncurses, meaning it wips the screen and sets up the background-color.
But this would give you the output of the process at least. replacing print(output.decode('UTF-8'))
would show you what that output is.
edited 20 hours ago
answered yesterday
Torxed
13k105286
13k105286
it looks that program stuck at this line of code """process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)""" after reaching this line the program start storing output in buffers and become silent .. the only thing i want to extract from ngrok while ngrok is running is the farwarded url that ngrok provides
– arslanmughal
yesterday
@arslanmughal You are partially correct, it's not stuck on thePopen()
line tho. It's stuck onprint(process.stdout.readline())
. This is a huge different because it just tells us thatPopen()
isn't able to get the output from the process. I also triedstdout.read(1)
which returns nothing. So I added an additional way of getting the output.
– Torxed
20 hours ago
add a comment |
it looks that program stuck at this line of code """process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)""" after reaching this line the program start storing output in buffers and become silent .. the only thing i want to extract from ngrok while ngrok is running is the farwarded url that ngrok provides
– arslanmughal
yesterday
@arslanmughal You are partially correct, it's not stuck on thePopen()
line tho. It's stuck onprint(process.stdout.readline())
. This is a huge different because it just tells us thatPopen()
isn't able to get the output from the process. I also triedstdout.read(1)
which returns nothing. So I added an additional way of getting the output.
– Torxed
20 hours ago
it looks that program stuck at this line of code """process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)""" after reaching this line the program start storing output in buffers and become silent .. the only thing i want to extract from ngrok while ngrok is running is the farwarded url that ngrok provides
– arslanmughal
yesterday
it looks that program stuck at this line of code """process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)""" after reaching this line the program start storing output in buffers and become silent .. the only thing i want to extract from ngrok while ngrok is running is the farwarded url that ngrok provides
– arslanmughal
yesterday
@arslanmughal You are partially correct, it's not stuck on the
Popen()
line tho. It's stuck on print(process.stdout.readline())
. This is a huge different because it just tells us that Popen()
isn't able to get the output from the process. I also tried stdout.read(1)
which returns nothing. So I added an additional way of getting the output.– Torxed
20 hours ago
@arslanmughal You are partially correct, it's not stuck on the
Popen()
line tho. It's stuck on print(process.stdout.readline())
. This is a huge different because it just tells us that Popen()
isn't able to get the output from the process. I also tried stdout.read(1)
which returns nothing. So I added an additional way of getting the output.– Torxed
20 hours ago
add a comment |
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53229305%2fdynamic-output-from-python-subprocess-module%23new-answer', 'question_page');
}
);
Post as a guest
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
2
Possible duplicate of Printing output in realtime from subprocess. Altho that's just on the question itself, but it won't work for ngrok or other ncurses application. So just leaving this here for others who end up here wondering how to get output from
subprocess
.– Torxed
20 hours ago