From: Martin Mueller <martinmueller@northwestern.edu>
To: "pgsql-general@postgresql.org" <pgsql-general@postgresql.org>
Subject: scaling up from t1n to 60 million records
Thread-Topic: scaling up from t1n to 60 million records
Thread-Index: AQHc55nMCqa1OUuiTk+CoVWmCRqQbA==
Date: Tue, 19 May 2026 14:27:28 +0000
Message-ID: 
 <CY8PR05MB1010861EAD48ED098786C9690C4002@CY8PR05MB10108.namprd05.prod.outlook.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: 
 =?us-ascii?Q?l9cNidr+q9Qc1ageRjgtfR6BN5B/UDMy+s2KpMH48wNr5BYNXw1KNVApFCKw?=
 =?us-ascii?Q?rDuAfBAIv3qd8/nVPSYxzhYr+iLFUp1gWiw9sFRyTPVZHbySnxLQIvWbzO8P?=
 =?us-ascii?Q?Nb7iM6MMQ1AqkN3syXGdgmJlC+FYtrxMacZPjgqeGKDOujFVedr/rf+namKk?=
 =?us-ascii?Q?KR6vBuQMGasooTh0Kfid8dymQMv6EfNY6QBUWFgQAIxYHnGNzjuJ1esQdMMC?=
 =?us-ascii?Q?Ti9Hf0+HUx8e4bAQnZHeuuYnrJXWWfkNuR/6uzTAjfozpIvm5ZRbzUAAndKa?=
 =?us-ascii?Q?cgpgcqV+dK5LG6yjGgMLgrxYjk9Z5K8l1K9D0oaJZ4vxg26BRzs6Uc/xzOqV?=
 =?us-ascii?Q?HHLUFgoKJfw5x2aQ68/0I0jfv4uDGHRAuprPNfZjXqTV65fKnV+/dmHnkiTr?=
 =?us-ascii?Q?u4hv7+Dg7pcmFLi1r6uYRsKfmZddurEl1RYXVvbL5upFoFLsiP/KL1Lteyk1?=
 =?us-ascii?Q?8/Vmb1qRX2oiCgBo+RNbcNhUDp0zzAfMOvLHaQgE6irVvwRQU3+HVOQm6RMM?=
 =?us-ascii?Q?PvrFVxtFBpFm6X43JY447GIiN1qmVjvZJYrhVW8aG5y5o9pZxSzb3H0nbW47?=
 =?us-ascii?Q?qF3JfsY4ThXcXUhp4ZjClN79aTtvdugAmjm9bxpfuoafjp3U5F2L30kXwt/l?=
 =?us-ascii?Q?ysDopNjqt2+IaPgAWsAaP8GEYQPFB0odeSOaF0vNkG/B7u1vq6EiamYgwLjR?=
 =?us-ascii?Q?Lz7LFCgPIXX2hX24BQiaS0sOOfdvhN23IkXyGkyFo0Tual+mnRvYvLiGkT9U?=
 =?us-ascii?Q?afKXULK2agWz7IviN1t8aPxAuOteXi8ptlmmtuS+C3pq2gQX8BhCi86OKmcF?=
 =?us-ascii?Q?NdjTgsr+SgchoUnQZgKX1fHtZzldtLHFeZ4C4m/fxAmuNoQTUZiJCm1WFAhF?=
 =?us-ascii?Q?iJJi9jneLgMjEghUNG2qs2plotJcxYJIOrwHvMOSb3YY0VNc98w5I3e/f5XS?=
 =?us-ascii?Q?BWrBQAa1XAkWPzZmVRsMT5fo4KI4gN+FEo2JYvq7uf59je8KEamUM6OPymih?=
 =?us-ascii?Q?QZmbMp/uLKV0IDcqSFlxQAHaKDuFFM6mKixpGbUsCEDaOk2rd82j/9OrzQqY?=
 =?us-ascii?Q?pouhCIxZyqUs+vKz/u0VkrESzXTwVVa4LJRKONyNPAYQLv1sw+MgjzhZunEQ?=
 =?us-ascii?Q?RotY6K0DCWGx47EDGO8A1LncpXusDtVB3fUSE9VZHcGAOhari2YDmPMpmv0v?=
 =?us-ascii?Q?C1Dl+O6YMLglU4q6wSm0Wi0+51bZoz/8N2IzofWFptcc6ievFP71WQmKBhQC?=
 =?us-ascii?Q?/Smu5JKRLRdezcjF0m84GgpF6aCl73enrZXVNsj3+yQmbSuDJAY9oyjbkY/j?=
 =?us-ascii?Q?Ngqb0nWIEsIhjxI27krvgF+7AXg1UHm/Y8qjYf7uivu1rPK9c0urRVJAD9Z1?=
 =?us-ascii?Q?MZH0j/lX28hErTX4KtGs3snvv+xrXJGA8cSM8fv+MJeoyS0PtMvh+2KgOuq9?=
 =?us-ascii?Q?7uGKrZGlf1IzjOFOeE1lj0VmkbPYS6rh2MvMtV/uRzdR/+g4PVDbUwFG3OIM?=
 =?us-ascii?Q?iOiwcLmjTmO0M90BrldJyduSoRhjvQalZ/9OnWCk8emaftz+W7RZM3x/7kBy?=
 =?us-ascii?Q?GKqkZ5FtyMyxtGgE2KT2qRqXBgQqE+AO+nRoHR2AoJGu5sCNdVA3dPHu16po?=
 =?us-ascii?Q?dN0FXDwufYTm6hdF5biqx56FKWCFFx5hSAO5v+rIULQvDO7alzN7O/B7qkzp?=
 =?us-ascii?Q?o8f4JEThg0eBEPgw6EZk6N3DllQTq17yO85t8tt3jJO8+wNeOsMH+IMxomVO?=
 =?us-ascii?Q?YknUInBvLy+cv9PMN+v4UkaJpI+sLYA3vMvOL5ZIh09jQu+pCuDc?=
Content-Type: multipart/alternative;
	boundary="_000_CY8PR05MB1010861EAD48ED098786C9690C4002CY8PR05MB10108na_"
MIME-Version: 1.0
X-Exchange-RoutingPolicyChecked: 
	Ob1U5LqwEtQGPlf8BVNJVYL8p/VQ5HKTICE25etsBo2fcE5RWVLqmxgjcFaBLOPue2SuO9rPu+x4f/Dcp2YphhmfFXyeF8FyVpeSK0w6tFqRRGdNL9C0sEajPmIyapLMDrkRCWpwOhqMSaTxqKZO7+vYUulWpFWDiH7YUo45oi5iq1IkHqmH3juI4Zua4yWODm3pIofmOW/6MDfW2YWze2EMS8lquudvPWcJSTejsz3iSPkgXbAy71qwy9Ssgst9jJ4PV/lcnUz6oMxWHP/3EzWK6+jp5qjYugWxgXcX4js8WZKn3A4ON+jdYUfR0pR/eMLlfFSawIr0YKS9bMGNyA==
X-OriginatorOrg: northwestern.edu
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: CY8PR05MB10108.namprd05.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 
 de8e966b-7ef2-4684-0e20-08deb5b2c10c
X-MS-Exchange-CrossTenant-originalarrivaltime: 19 May 2026 14:27:28.0267
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 7d76d361-8277-4708-a477-64e8366cd1bc
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: 
 LShPZetclRrS9EuDo86k9U8q9AIZAdNw3LXhiHCeEies+xni+QmaGgbzRpI7QV/SYdgAOEyOUZfKEnmH1Bqa4C8DD7sKFzMWrXwpmxMv5Zs=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA3PR05MB10419
X-Proofpoint-GUID: lkGJYI_OGzgepi4YgiDX-rfluKStvWyZ
X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwNTE5MDE0MiBTYWx0ZWRfX7qPL8P+2eFxu
 s1Aec8n7Jrb3VByK0GF0ej/s4bcKPYAzKLQhjlTS1YnVoFgKhQXMNC5e/lcmAbwF7Jq3yLBhe0q
 jSJz0sgCA2rP/QfQCRcvmSQA4QdGl4rx6wOWq/ksX7X/kXjSg64ClpSGOjK00kQizNtx2MAu25A
 vP+8g/rtmZXsnbDZOqIpNi0HN1v0G1DJvr/7GkQe/XkSRx/+ZyyyoH1y6ocjlhVPKxv8lmJ32O5
 9kYYlLSBjB9JjACGpJR0uDitvdUKAp8D/mcTF1QLv4i26vdiKoz29q9JgJDkEYzZkMcIi+O51XF
 6/YI3EjCH14aIar+Vxyahj7OObl6d6rfgsc4MFdV5d+myHhVZGKgE6a97g8t054QiYsiNCnhzsh
 wrn/7Inl
X-Proofpoint-ORIG-GUID: lkGJYI_OGzgepi4YgiDX-rfluKStvWyZ
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.293,Aquarius:18.0.1143,Hydra:6.1.51,FMLib:17.12.100.49
 definitions=2026-05-19_03,2026-05-18_01,2025-10-01_01
X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0
 suspectscore=0 lowpriorityscore=0 mlxlogscore=828 mlxscore=0 adultscore=0
 bulkscore=0 priorityscore=1501 malwarescore=0 impostorscore=0 clxscore=1011
 spamscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.19.0-2605130000 definitions=main-2605190142
List-Id: <pgsql-general.lists.postgresql.org>
List-Help: <https://lists.postgresql.org/manage/>
List-Subscribe: <https://lists.postgresql.org/manage/>
List-Post: <mailto:pgsql-general@lists.postgresql.org>
List-Owner: <mailto:pgsql-general-owner@lists.postgresql.org>
List-Archive: <https://www.postgresql.org/list/pgsql-general>
Archived-At: 
 <https://www.postgresql.org/message-id/CY8PR05MB1010861EAD48ED098786C9690C4002%40CY8PR05MB10108.namprd05.prod.outlook.com>
Precedence: bulk

--_000_CY8PR05MB1010861EAD48ED098786C9690C4002CY8PR05MB10108na_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

I use Postgres with a GUI frontend (Aquafold) as a very large spreadsheet o=
n steroids that analyzes rare or defective spellings in a corpus of 65,000 =
texts and1.5 billion words.  I typically extract  data from the corpus with=
 python scripts, turn them into tables and load them into the database.

On my Mac with 32 GB of memory performance is OK with queries that typicall=
y within seconds extract data rows from tables  with up to ten million rows=
.  If the result set is large, I suspect that most of time machine's time i=
s spent displaying result sets. I have used indexing sparingly. While it he=
lps, the time savings often don't matter much.

I am thinking about scaling up to table with about 60 million rows.  Are th=
ere things to do or watch out for? Or should I proceed on the assumption th=
at that 60 million records are within scope and that the added timecost is =
roughly linear?

Martin Mueller
Professor emeritus of English and Classics
Northwestern University


--_000_CY8PR05MB1010861EAD48ED098786C9690C4002CY8PR05MB10108na_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
>
</head>
<body>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-family: Aptos, sans-serif=
; font-size: 12pt;">
<span style=3D"font-size: 14pt; color: rgb(0, 0, 0);">I use Postgres with a=
 GUI frontend (Aquafold) as a very large spreadsheet on steroids that analy=
zes rare or defective spellings in a corpus of 65,000 texts and1.5 billion =
words. &nbsp;I typically extract &nbsp;data
 from the corpus with python scripts, turn them into tables and load them i=
nto the database.</span></p>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-family: Aptos, sans-serif=
; font-size: 12pt;">
<span style=3D"font-size: 14pt; color: rgb(0, 0, 0);"><br>
</span></p>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-family: Aptos, sans-serif=
; font-size: 12pt;">
<span style=3D"font-size: 14pt; color: rgb(0, 0, 0);">On my Mac with 32 GB =
of memory performance is OK with queries that typically within seconds extr=
act data rows from tables &nbsp;with up to ten million rows. &nbsp;If the r=
esult set is large, I suspect that most of time
 machine's time is spent displaying result sets. I have used indexing spari=
ngly. While it helps, the time savings often don't matter much.&nbsp;</span=
></p>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-family: Aptos, sans-serif=
; font-size: 12pt;">
<span style=3D"font-size: 14pt; color: rgb(0, 0, 0);"><br>
</span></p>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-family: Aptos, sans-serif=
; font-size: 12pt;">
<span style=3D"font-size: 14pt; color: rgb(0, 0, 0);">I am thinking about s=
caling up to table with about 60 million rows. &nbsp;Are there things to do=
 or watch out for? Or should I proceed on the assumption that that 60 milli=
on records are within scope and that the
 added timecost is roughly linear?</span></p>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-family: Aptos, sans-serif=
; font-size: 12pt;">
&nbsp;</p>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-family: Aptos, sans-serif=
; font-size: 12pt;">
<span style=3D"font-family: Calibri, sans-serif; font-size: 14pt;">Martin M=
ueller</span></p>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-family: Aptos, sans-serif=
; font-size: 12pt;">
<span style=3D"font-family: Calibri, sans-serif; font-size: 14pt;">Professo=
r emeritus of English and Classics</span></p>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-family: Aptos, sans-serif=
; font-size: 12pt;">
<span style=3D"font-family: Calibri, sans-serif; font-size: 14pt;">Northwes=
tern University</span></p>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-family: Aptos, sans-serif=
; font-size: 12pt;">
<span style=3D"font-size: 14pt;">&nbsp;</span></p>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-family: Aptos, sans-serif=
; font-size: 12pt;">
<span style=3D"font-size: 14pt;">&nbsp;</span></p>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-family: Aptos, sans-serif=
; font-size: 12pt;">
<span style=3D"font-size: 14pt;">&nbsp;</span></p>
</body>
</html>

--_000_CY8PR05MB1010861EAD48ED098786C9690C4002CY8PR05MB10108na_--