Optimizing large data storing - TinyDB?

Hello Koders :smile:
I am facing a problem I couldn’t fix my self after a lot of testing, I have a ‘csv’ file containing more than 7500+ row and each raw have two values.

my problem is when ever I try to store this list into a TinyDB database my app freezes for minutes and becomes unresponsive, I’ve used these blocks to store the list.
blocks

any help/suggestion is appreciated :slightly_smiling_face:

As you said, there is 7.5k+ row, which means the loop has 7 thousand tags & values to store. Is there a reason you need to do that?

It is not the tiny DB freezes, but the loop can’t handle that much in a short period.

@WatermelonIce Hi, I am aware of that, my whole app is based on this huge list and I am seeking for a way to optimize the storing method :smile:

I’ve used Clock component interval to store each 500 row from total rows with no success.

Thank you!

I would use online cloud platform like firebase.
However, if it is an offline app, just use file component.
If you use clock, still, it needed a long time to run.

Thank you for your contribution to this post but I am sure there is a solution out there and I hope someone can shed some light on this topic :grinning:

PS: the app is offline.

Why not just store it as a list?
Or, you can make it a dictionary.
Read this guide to know to use of dictionary and tinyDB by me:

1 Like

@WatermelonIce this is a great topic!

I will test it shortly and if it solved my problem I will mark it as solution!

Thank you :grin:

I had no luck using Dictionary component :cry:

I spent the whole day working on this problem, I’ve tried everything I know so far with no lock.

finally I am forced to use a list variable like I did the first time and as you suggested.

Thank you @WatermelonIce

alternatively you might want to use the sqlite database
https://docs.kodular.io/components/storage/sqlite/
Taifun

Hi @Taifun
I will give it a try but does it have any limitations? like query limit for example, excuse my ignorance.

This way will not freeze the app but will take more time.

2 Likes

Unbenannt
taken from Marilyn Monroe Quote: “The sky is not the limit. Your mind is.” | Rilke quotes, Arthur schopenhauer quotes, Murakami quotes

you might want to read more here What are the limitations of SQLite - بحث Google‏
Taifun

1 Like

Thank you very much @Taifun but @Hassan provided a simple great solution to my problem.

Hi @Hassan
Thank you very much :grinning: I’ve been struggling for days for this solution with no lock, and here you are provided a great solution to my problem, Thank you again and again :grin: :smiling_face_with_three_hearts:

All the records -total=7887- was successfully saved in TinyDB with no lag and no freeze and in about 100 seconds!! :astonished:

Thank you @WatermelonIce
Thank you @Taifun

2 Likes

another solution is to convert the csv table into a list of lists using the list from csv atble block and store the complete list of lists in only one tag in TinyDB
then later if you like to get a value, use the lookup in pairs block
How does the lookup in pairs block work?

Taifun

1 Like

This will be handy :grinning: Thank you @Taifun

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.