Re: scaling up from t1n to 60 million records

public inbox for pgsql-general@postgresql.org  
help / color / mirror / Atom feed

From: Jan Karremans <karremans.ja@gmail.com>
To: Martin Mueller <martinmueller@northwestern.edu>
Cc: pgsql-general@postgresql.org <pgsql-general@postgresql.org>
Subject: Re: scaling up from t1n to 60 million records
Date: Tue, 19 May 2026 16:32:39 +0200
Message-ID: <ADA1E0D9-0DA4-4E90-8596-D04DBFD20E4B@gmail.com> (raw)
In-Reply-To: <CY8PR05MB1010861EAD48ED098786C9690C4002@CY8PR05MB10108.namprd05.prod.outlook.com>
References: <CY8PR05MB1010861EAD48ED098786C9690C4002@CY8PR05MB10108.namprd05.prod.outlook.com>

Dear Martin,

I think you would be mostly good for just going ahead with this.
You might look at the size of your tables, but I expect that all to be well within safe ranges.

Cheers,
Jan

> On 19 May 2026, at 16:27, Martin Mueller <martinmueller@northwestern.edu> wrote:
> 
> I use Postgres with a GUI frontend (Aquafold) as a very large spreadsheet on steroids that analyzes rare or defective spellings in a corpus of 65,000 texts and1.5 billion words.  I typically extract  data from the corpus with python scripts, turn them into tables and load them into the database.
> 
> On my Mac with 32 GB of memory performance is OK with queries that typically within seconds extract data rows from tables  with up to ten million rows.  If the result set is large, I suspect that most of time machine's time is spent displaying result sets. I have used indexing sparingly. While it helps, the time savings often don't matter much. 
> 
> I am thinking about scaling up to table with about 60 million rows.  Are there things to do or watch out for? Or should I proceed on the assumption that that 60 million records are within scope and that the added timecost is roughly linear?
>  
> Martin Mueller
> Professor emeritus of English and Classics
> Northwestern University
>  
>  
>

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-general@postgresql.org
  Cc: karremans.ja@gmail.com, martinmueller@northwestern.edu
  Subject: Re: scaling up from t1n to 60 million records
  In-Reply-To: <ADA1E0D9-0DA4-4E90-8596-D04DBFD20E4B@gmail.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox