Newer
Older
til / python / 20231117_save_resultset_to_file.html
@clewis clewis on 17 Nov 2023 1 KB python - save/read resultsets

<h3>
Save ResultSet to a File
</h3>

<p>
At work I had a huge query with many inner joins and also an inner query that
was taking forever and wouldn't always return, it would error out
instead. (Turns out the main problem was hitting a very active DB but when I
switched to a backup DB the query worked fine.)
</p>

<p>
Anyways, in order not to worry about making the query whenever I wanted to test
something I decided to save the Python resultset to a file.  I wanted to save it
in such a way as that when I read it back in it was as if I was using the
resultset.
</p>

<p>
Pickle didn't seem to "easily" work, i.e. after a few tries I just gave up on
it.  But after some Googling I found this solution which is just what I wanted.
</p>

<p><b>Execute query, fetch data into a resultset</b></p>
<code>
results = cursor.execute(some_query)
rows = results.fetchall()
</code>

<p><b>Convert the result set to tuples and save</b></p>
<pre>
toops = [tuple(row) for row in rows]

with open('some_file.dat', 'w') as f:
    for tup in toops:
        f.write(f'{tup}\n')
</pre>

<p><b>Read data back in</b></p>
<pre>
toops = []
with open('some_file.dat') as f:
    for line in f.readlines():
        toops.append(eval(line))
</pre>

<p><b>Note:</b></p>
<p>If the data has any python datetime objects or Decimal objects then you will
  need to import the following:
</p>

<pre>
import datetime
from decimal import *
</pre>

<p>
  You can now use <code>toops</code> as if it were <code>toops =
  results.fetchall()</code>
</p>