Optimizing large data storing - TinyDB?

[Silver (Silver SY)]

Hello Koders
I am facing a problem I couldn’t fix my self after a lot of testing, I have a ‘csv’ file containing more than 7500+ row and each raw have two values.

my problem is when ever I try to store this list into a TinyDB database my app freezes for minutes and becomes unresponsive, I’ve used these blocks to store the list.

any help/suggestion is appreciated

[WatermelonIce (Bryan)]

As you said, there is 7.5k+ row, which means the loop has 7 thousand tags & values to store. Is there a reason you need to do that?

It is not the tiny DB freezes, but the loop can’t handle that much in a short period.

[Silver (Silver SY)]

@WatermelonIce Hi, I am aware of that, my whole app is based on this huge list and I am seeking for a way to optimize the storing method

I’ve used Clock component interval to store each 500 row from total rows with no success.

Thank you!

[WatermelonIce (Bryan)]

I would use online cloud platform like firebase.
However, if it is an offline app, just use file component.
If you use clock, still, it needed a long time to run.

[Silver (Silver SY)]

@WatermelonIce

I would use online cloud platform like firebase.
However, if it is an offline app, just use file component.
If you use clock, still, it needed a long time to run.

Thank you for your contribution to this post but I am sure there is a solution out there and I hope someone can shed some light on this topic

PS: the app is offline.

[WatermelonIce (Bryan)]

Why not just store it as a list?
Or, you can make it a dictionary.
Read this guide to know to use of dictionary and tinyDB by me:

Simple use of dictionary and TinyDB

Introduction Have you ever use tinyDB or other databases to store value? When there are so many data you have to store, you won’t want to store it with so many call tinyDB store block. Therefore, the dictionary can be used to help. What is a dictionary? Dictionary is a built-in component. Generally speaking, a dictionary uses a key to call and get the value. For example, It will return the string "whatever". [blocks (2)] How to use it? As mentioned, a dictionary uses a key to call and ge…

[Silver (Silver SY)]

@WatermelonIce this is a great topic!

I will test it shortly and if it solved my problem I will mark it as solution!

Thank you

[Silver (Silver SY)]

@WatermelonIce

Why not just store it as a list?
Or, you can make it a dictionary.
Read this guide to know to use of dictionary and tinyDB by me:
Simple use of dictionary and TinyDB

I had no luck using Dictionary component

I spent the whole day working on this problem, I’ve tried everything I know so far with no lock.

@WatermelonIce

Why not just store it as a list?

finally I am forced to use a list variable like I did the first time and as you suggested.

Thank you @WatermelonIce

[Taifun]

@WatermelonIce

Why not just store it as a list?
Or, you can make it a dictionary.

alternatively you might want to use the sqlite database
https://docs.kodular.io/components/storage/sqlite/
Taifun

[Silver (Silver SY)]

@Taifun

@WatermelonIce

Why not just store it as a list?
Or, you can make it a dictionary.

alternatively you might want to use the sqlite database
SQLite - Kodular Docs
Taifun

Hi @Taifun
I will give it a try but does it have any limitations? like query limit for example, excuse my ignorance.

[Hassan]

This way will not freeze the app but will take more time.

[Taifun]

@Silver

I will give it a try but does it have any limitations? like query limit for example, excuse my ignorance.

taken from Marilyn Monroe Quote: “The sky is not the limit. Your mind is.” | Rilke quotes, Arthur schopenhauer quotes, Murakami quotes

you might want to read more here What are the limitations of SQLite - بحث Google‏
Taifun

[Silver (Silver SY)]

@Taifun

you might want to read more here What are the limitations of SQLite - بحث Google‏
Taifun

Thank you very much @Taifun but @Hassan provided a simple great solution to my problem.

[Silver (Silver SY)]

@Hassan

This way will not freeze the app but will take more time.

Hi @Hassan
Thank you very much I’ve been struggling for days for this solution with no lock, and here you are provided a great solution to my problem, Thank you again and again

All the records -total=7887- was successfully saved in TinyDB with no lag and no freeze and in about 100 seconds!!

Thank you @WatermelonIce
Thank you @Taifun

[Taifun]

@Silver

I have a ‘csv’ file containing more than 7500+ row and each raw have two values.

another solution is to convert the csv table into a list of lists using the list from csv atble block and store the complete list of lists in only one tag in TinyDB
then later if you like to get a value, use the lookup in pairs block
How does the lookup in pairs block work?

Taifun

[Silver (Silver SY)]

@Taifun

another solution is to convert the csv table into a list of lists using the list from csv atble block and store the complete list of lists in only one tag in TinyDB
then later if you like to get a value, use the lookup in pairs block
How does the lookup in pairs block work?

Taifun

This will be handy Thank you @Taifun