Click here to Skip to main content
15,445,242 members

Welcome to the Lounge

   

For discussing anything related to a software developer's life but is not for programming questions. Got a programming question?

The Lounge is rated Safe For Work. If you're about to post something inappropriate for a shared office environment, then don't post it. No ads, no abuse, and no programming questions. Trolling, (political, climate, religious or whatever) will result in your account being removed.

 
GeneralRe: That's not the way I remember that scene... Pin
CodeWraith20-Dec-20 21:44
MemberCodeWraith20-Dec-20 21:44 
PraiseI did it! Huge JSON on 8-bit w/ < 2kB of RAM Pin
honey the codewitch20-Dec-20 12:14
mvahoney the codewitch20-Dec-20 12:14 
GeneralRe: I did it! Huge JSON on 8-bit w/ < 2kB of RAM Pin
Randor 20-Dec-20 13:34
professional Randor 20-Dec-20 13:34 
GeneralRe: I did it! Huge JSON on 8-bit w/ < 2kB of RAM Pin
honey the codewitch20-Dec-20 13:39
mvahoney the codewitch20-Dec-20 13:39 
GeneralRe: I did it! Huge JSON on 8-bit w/ < 2kB of RAM Pin
Randor 20-Dec-20 14:04
professional Randor 20-Dec-20 14:04 
GeneralRe: I did it! Huge JSON on 8-bit w/ < 2kB of RAM Pin
honey the codewitch20-Dec-20 14:06
mvahoney the codewitch20-Dec-20 14:06 
GeneralRe: I did it! Huge JSON on 8-bit w/ < 2kB of RAM Pin
Randor 21-Dec-20 11:31
professional Randor 21-Dec-20 11:31 
GeneralRe: I did it! Huge JSON on 8-bit w/ < 2kB of RAM Pin
honey the codewitch21-Dec-20 11:54
mvahoney the codewitch21-Dec-20 11:54 
Nice. Right now I'm thrilled with myself, because after several innovations I was able to cram query results into dozens of bytes per row (and I only needed one row at a time) - basically the JSON reader *only* needs space enough in the capture for the largest string you're actually going to examine - you skip everything else and scan denormalized through the document until you find what you want, then you "extract" it - extractions are extremely efficient nested structures that allow you to direct the pull parser to retrieve certain elements. That way you can for example, retrieve say, 3 relevant fields off of an object regardless of field order, and one of those fields could have subobjects or arrays that get values extracted from them.
It does this all in one pass, and then returns the reader to a known good position after it has read over the necessary bit of the document. It returns tight structures representing all the data it captured in highly efficient recyclable memory pools which you can free any time you don't need anymore at almost no cost.

...(clipped)
S07E09 Bitter Pill
S07E10 Things Unseen
S07E11 Tipping Point
S07E12 Sea Change
S07E13 Reckoning
Scanned 112 episodes and -5320 characters in 9425 milliseconds using 23 bytes of LexContext and 50 bytes of the pool


Forgive the negative value - the little 8 bit proc can't handle a number in the 190k range being dumped to the serial port. it overflows and goes negative is all. otherwise everything on there is accurate. It took a total (not counting incidental locals on the stack) 73 bytes to extract an episode_number, season_number and name from each of the 112 episodes in the 200k burn notice series JSON document.

The reason it matters is this source JSON document I'm using is actually composed of several very "chunky" online dumps of the sort you tend to get from mongoDB repositories. I got the dumps from tmdb.com. Each http request generates lots of data, and you have to pick through it, which is what this is ultimately for - but it works on files too.
Real programmers use butterflies

GeneralRe: I did it! Huge JSON on 8-bit w/ < 2kB of RAM Pin
Randor 21-Dec-20 12:06
professional Randor 21-Dec-20 12:06 
GeneralRe: I did it! Huge JSON on 8-bit w/ < 2kB of RAM Pin
honey the codewitch21-Dec-20 13:03
mvahoney the codewitch21-Dec-20 13:03 
GeneralRe: I did it! Huge JSON on 8-bit w/ < 2kB of RAM Pin
Randor 21-Dec-20 13:47
professional Randor 21-Dec-20 13:47 
GeneralRe: I did it! Huge JSON on 8-bit w/ < 2kB of RAM Pin
honey the codewitch21-Dec-20 13:56
mvahoney the codewitch21-Dec-20 13:56 
JokeWho is Santa’s favorite singer? Pin
Sandeep Mewara20-Dec-20 7:24
mveSandeep Mewara20-Dec-20 7:24 
GeneralRe: Who is Santa’s favorite singer? Pin
W Balboos, GHB20-Dec-20 8:33
mveW Balboos, GHB20-Dec-20 8:33 
GeneralRe: Who is Santa’s favorite singer? Pin
OriginalGriff20-Dec-20 9:21
mveOriginalGriff20-Dec-20 9:21 
GeneralNew Normal Christmas songs Pin
OriginalGriff20-Dec-20 4:27
mveOriginalGriff20-Dec-20 4:27 
GeneralRe: New Normal Christmas songs Pin
W Balboos, GHB20-Dec-20 4:55
mveW Balboos, GHB20-Dec-20 4:55 
GeneralRe: New Normal Christmas songs Pin
RickZeeland20-Dec-20 5:38
mveRickZeeland20-Dec-20 5:38 
GeneralRe: New Normal Christmas songs Pin
PIEBALDconsult20-Dec-20 6:16
professionalPIEBALDconsult20-Dec-20 6:16 
GeneralRe: New Normal Christmas songs Pin
W Balboos, GHB20-Dec-20 6:34
mveW Balboos, GHB20-Dec-20 6:34 
GeneralRe: New Normal Christmas songs Pin
Richard MacCutchan20-Dec-20 7:11
mveRichard MacCutchan20-Dec-20 7:11 
GeneralRe: New Normal Christmas songs Pin
W Balboos, GHB20-Dec-20 7:15
mveW Balboos, GHB20-Dec-20 7:15 
GeneralRe: New Normal Christmas songs Pin
Richard MacCutchan20-Dec-20 7:21
mveRichard MacCutchan20-Dec-20 7:21 
GeneralRe: New Normal Christmas songs Pin
Daniel Pfeffer20-Dec-20 8:07
professionalDaniel Pfeffer20-Dec-20 8:07 
GeneralRe: New Normal Christmas songs Pin
Nelek20-Dec-20 8:28
protectorNelek20-Dec-20 8:28 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.


Straw Poll

What topics would you like to learn more about?
Contributed by Marc Clifton.
  Results   532 votes