I'd just like to get some experienced software engineers' input on this situation.
I had a somewhat large dataset of nested JSON information, and I had a task of processing these JSONs with Python and inputting this information into a PostgreSQL database.
The difficulty came in implementing this efficiently. Parsing and structuring the JSON information itself took a while. And inserting the entries into the database took longer.
I basically had to parse through 6 levels of nesting, and used many nested for loops to get the desired information and structure it in a manner appropriate for the database.
For the dataset I had, this was manageable, thought database creation took about 5 minutes (for an insertion of over a million lines of code).
As you can imagine, my concern was that this implementation is absolutely not scalable.
So my questions are 1) How do you deal with processing JSON information for databases and 2) How to deal with nested JSONs, specifically
I'd appreciate any insight :)
Aucun commentaire:
Enregistrer un commentaire