[python-sybase] memory usage
andrewjt at applecomm.net
Mon, 20 Dec 2004 12:22:11 +1100
I did notice another post about high memory usage. I too am experiencing
the same thing. However eventually my machines runs out of physical and
swap memory and then the script is terminated! :(
Basically I have a script that is parsing around 2,000,000 rows of data.
For every row, a field is extracted and then passed to a stored
procedure to get some information back.
Watching the process in top, I can see the memory increasing steadily.
If I take out the Sybase stored procedure stuff and just hard code a
value of the stored procedure output, the script runs to completion
never using more than 4mb of memory.
However with the Sybase query in, memory just steadily increases.
PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU
89567 ajt 112 0 21604K 19320K RUN 1:42 28.81% 28.81%
I'm running my script on Freebsd 5.3 using the port: py24-sybase-0.36_1
My script does the following:
# connect to db
db = Sybase.connect('dbhost', 'user', 'pass', 'db')
# start for loop
for text in fileinput.input("datafile"):
# get cursor
mnpcursor = db.cursor()
# run stored proc
# get output
row = mnpcursor.fetchone ()
# do stuff with row
# close cursor
Is there are better way to be doing this or something I'm missing?