agora inbox for postgres@postgres.berkeley.edu  

help / color / mirror / Atom feed

From: Paul M. Aoki <aoki@postgres.Berkeley.EDU>
To: Mark Victor Priester <priester@student.msu.edu>
Cc: postgres@postgres.Berkeley.EDU
Subject: Re: Another import question
Date: Tue, 12 Apr 94 10:56:14 -0700
Message-ID: <199404121756.KAA16577@faerie.CS.Berkeley.EDU> (raw)
In-Reply-To: <9404121412.AA78322@student4.cl.msu.edu>

Mark Victor Priester <priester@student.msu.edu> writes:
> 1. This works fine for 10 instances/lines even for 100 instances,
>    but when I set it to work on the whole file (~1.5M instances)
>    it chokes early on in the game.

postgres "keeps" too much stuff lying around from command to
command during a single transaction.  you can think of this as 
a memory leak, but it does get cleaned up at end-of-transaction,
so memory leak isn't quite the right way to describe it.  so
running large numbers of commands in a single transaction
(on the order of 10000, depending on how much swap you have) 
causes you to run out of memory.

so: you can load in smaller wads.

since you are turning the values into ascii anyway (so that
you can use "append") it seems like you could just create a 
file in the "copy" (tab-delimited) format and use "copy from".
this is far more efficient than using append commands (no
query parsing/optimization overhead, runs as a single xact).
--
  Paul M. Aoki  |  CS Div., Dept. of EECS, UCB  |  aoki@postgres.Berkeley.EDU
                |  Berkeley, CA 94720           |  ...!uunet!ucbvax!aoki

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: postgres@postgres.berkeley.edu
  Cc: aoki@postgres.Berkeley.EDU, priester@student.msu.edu
  Subject: Re: Another import question
  In-Reply-To: <199404121756.KAA16577@faerie.CS.Berkeley.EDU>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox