Optimizing large data storing - TinyDB?

Hello Koders :smile:
I am facing a problem I couldn’t fix my self after a lot of testing, I have a ‘csv’ file containing more than 7500+ row and each raw have two values.

my problem is when ever I try to store this list into a TinyDB database my app freezes for minutes and becomes unresponsive, I’ve used these blocks to store the list.

any help/suggestion is appreciated :slightly_smiling_face:

As you said, there is 7.5k+ row, which means the loop has 7 thousand tags & values to store. Is there a reason you need to do that?

It is not the tiny DB freezes, but the loop can’t handle that much in a short period.

@WatermelonIce Hi, I am aware of that, my whole app is based on this huge list and I am seeking for a way to optimize the storing method :smile:

I’ve used Clock component interval to store each 500 row from total rows with no success.

Thank you!

I would use online cloud platform like firebase.
However, if it is an offline app, just use file component.
If you use clock, still, it needed a long time to run.

Thank you for your contribution to this post but I am sure there is a solution out there and I hope someone can shed some light on this topic :grinning:

PS: the app is offline.

Why not just store it as a list?
Or, you can make it a dictionary.
Read this guide to know to use of dictionary and tinyDB by me:

1 Like

@WatermelonIce this is a great topic!

I will test it shortly and if it solved my problem I will mark it as solution!

Thank you :grin:

I had no luck using Dictionary component :cry:

I spent the whole day working on this problem, I’ve tried everything I know so far with no lock.

finally I am forced to use a list variable like I did the first time and as you suggested.

Thank you @WatermelonIce

alternatively you might want to use the sqlite database

Hi @Taifun
I will give it a try but does it have any limitations? like query limit for example, excuse my ignorance.

This way will not freeze the app but will take more time.


taken from Marilyn Monroe Quote: “The sky is not the limit. Your mind is.” | Rilke quotes, Arthur schopenhauer quotes, Murakami quotes

you might want to read more here What are the limitations of SQLite - بحث Google‏

1 Like

Thank you very much @Taifun but @Hassan provided a simple great solution to my problem.

Hi @Hassan
Thank you very much :grinning: I’ve been struggling for days for this solution with no lock, and here you are provided a great solution to my problem, Thank you again and again :grin: :smiling_face_with_three_hearts:

All the records -total=7887- was successfully saved in TinyDB with no lag and no freeze and in about 100 seconds!! :astonished:

Thank you @WatermelonIce
Thank you @Taifun


another solution is to convert the csv table into a list of lists using the list from csv atble block and store the complete list of lists in only one tag in TinyDB
then later if you like to get a value, use the lookup in pairs block
How does the lookup in pairs block work?


1 Like

This will be handy :grinning: Thank you @Taifun

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.