How many documents are in your Paperless-ngx instance?

[nodecentral]

Hi,

As I continue to load more and more documents into my Paperless instance, I started to wonder if it had any limits etc.? I expect much of that will be down to (a) the performance you want, (b) what you’re running it on, and (c) how you’ve set it up to run..

So for those willing to share their numbers and set up - I’ll kick things off with mine…

Document Count = 2,303
Storage Facility (dB) = Postgres
Memory Facitlty = Redis
Install Method = Docker
Host/Hardware = QNAP NAS 453D (16GB)

[sbrunner]

Document Count = 7 800
Storage Facility (dB) = Postgres
Memory Facitlty = Redis
Install Method = Docker
Host/Hardware = Synology DS918+

[sonnyka]

Document Count = 2500
Storage Facility (dB) = Postgres
Memory Facitlty = Redis
Install Method = Docker
Host/Hardware = Synology DS218+

(would love to migrate to MySQL/MariaDB)

[nalpo]

Document Count = 370
Storage Facility (dB) = Postgres
Memory Facitlty = Redis
Install Method = Docker
Host/Hardware = Intel Nuc

I am curious: How do you handle classification and titling with that many documents, do you use any automatisms?

[sbrunner]

Yes, I put all the tag and correspondent to automatize, and notes that I have about 15 years of classified documents :-)

[titro]

Document Count = 1,952
Storage Facility (dB) = Postgres
Memory Facitlty = Redis
Install Method = Docker
Host/Hardware = Raspberry Pi 4, 4 GB RAM

[Sky-Dragon]

Document Count = first 300 (more below) now, 23 classified, 194 INBOX and >3000 to import
Storage Facility (dB) = Postgres
Memory Facitlty = Redis
Install Method = Docker
Host/Hardware = Raspberry Pi 3, Model B (aarch64) - 1 GB RAM
Host-OS: DietPi
Size of classification_model.pickle: 254 MB

Because my old classification concept was not the best, so I started new (hope it will be better now) - maybe someone can share his approach?

The main problem was that I couldn't classify any documents anymore, with about 50 classified documents the classification_model.pickle had over 800 MB so the container always restarted when I wanted to classify a document, I guess the RAM of the RPi 3 is just not enough.
On my "(test) import" WSL I could still do everything.
Now the classification_model.pickle has a size of 254 MB with 27 documents, how big is yours?

[nalpo]

I am at roughly 110MB atm. Do you use "auto" for every tag you have?

[Sky-Dragon]

95% of the time I use "auto", only a few tags/correspondents are on manual.

[virtadpt]

Document Count = 519
Storage Facility (dB) = MySQL
Memory Facilty = Redis
Install Method = Bare metal
Host/Hardware = Arch Linux, 24 cores, 64 GB RAM, 48 TB RAID-1
Size of classification_model.pickle: 77MB

This is a brand new install that I'm just starting to use heavily.

[Paddy4711]

Document Count = 1662
Storage Facility (dB) = Postgres
Memory Facilty = Redis
Install Method = Docker
Host/Hardware = Raspberry Pi 4 / 4GB RAM
Size of classification_model.pickle: 50MB
correspondents = 230
tags = 73
document types = 159

Works like a charm. The majority tagging / correspondents / document type is set to "auto", no problems at all so far!

[nodecentral]

i’m liking the new Statistics view on the main page..

[braberli]

Document Count = 13987
Storage Facility (dB) = Mariadb
Memory Facilty = Redis
Install Method = Docker
Hardware/Host = 64-bit, 4 CPU, Intel Core i3-3120M CPU @2.50GHz, 4G RAM
Size of classification_model.pickle: 600MB
correspondents = 1294
tags = 246
document types = 46

The great automatic classifier does not work anymore as it cannot be generated due to the size (see #2434).
The majority tagging / correspondents / document type is set to "auto". Problems occured at approx. more than 8.000 documents

[shivaRamdeen]

Do you have search performance issues with this? I have 2k documents 4 cpu, 8 gb ram and cannot do more than 1 search at a time and its very slow, 60s+, trying to debug maybe a config issue.

[kamilkosek]

The first batch, for now only the most recent documents from 2022-now , got imported:

Some words on the hardware:
Debian VM with 32GB RAM, 32 Cores , 8 Disk NVMe ZFS Storage
The import took about 26h with an import rate at approx 6k Docs/hour.

Classifier stats as for now: 673 documents, 18 tag(s), 150 correspondent(s), 24 document type(s). 1 storage path(es), it takes ~18s.

The biggest issue we are facing now, is the WebApp. Sometimes it just behaves very odd, like you do a search, open an "edit document" view, then out of nowhere, the edit view closes and the search results are displayed again. I think it sources from the impatience from the users and the are just clicking somewhere as soon as the search result does not appear immedietaly. One thing which could help, would be a different overlay (not displaying any documents) for the document list, as soon as a search is in progress.

I will keep you updated about our journey 🤘🏽

[nodecentral]

Wow, and all this is on a VM running docker, Redis and mariadb ?

[kamilkosek]

Yeah,
docker, redis and Postgres.
Postgres config was a bit modified with the help of pgtune

The current classifier stats:

[2023-08-25 09:05:00,694] [DEBUG] [paperless.classifier] Gathering data from database...
[2023-08-25 09:05:04,793] [DEBUG] [paperless.classifier] 1628 documents, 21 tag(s), 232 correspondent(s), 26 document type(s). 1 storage path(es)
[2023-08-25 09:05:04,793] [DEBUG] [paperless.classifier] Vectorizing data...
[2023-08-25 09:05:13,700] [DEBUG] [paperless.classifier] Training tags classifier...
[2023-08-25 09:05:18,194] [DEBUG] [paperless.classifier] Training correspondent classifier...
[2023-08-25 09:05:23,468] [DEBUG] [paperless.classifier] Training document type classifier...
[2023-08-25 09:05:26,703] [DEBUG] [paperless.classifier] Training storage paths classifier...

Actually I am facing an issue with the UI: #4054

[almereyda]

Impressive numbers. Would be interesting to derive a test case for this kind of setups, so nuisances and optimisations could be reproduced for development.

[flemmingss]

Document Count = 483
Storage Facility (dB) = MaiaDB
Memory Facitlty = Redis
Install Method = Docker
Host/Hardware = UnRAID, (128GB/2xIntel Xeon CPUs)

[github-actions[bot]]

This discussion has been automatically closed due to inactivity. Please see our contributing guidelines for more details.