In a synchronous runtime (say c# ) I can write a very simple recursive descent parser for a file that can output results as it reads the file. So I never need to buffer the file. There can be a complicated state machine pulling from the input, and it's nice and simple.
Something like this
BinaryStream s = File.Open("somefile");
while(s.CanRead() ){
var type = s.readInt32();
if ( type == 1 ) {
emit(readSomeBinaryStructure(s);
}else if ( type == 2 ){
emit ( readSomeOtherBInaryStructure(s))
emit (s.readString() );
}else{
emit(type);
}
}
In Node I have something very complicated because I get chunks of data which I can't determine the alignment on a structure.
What patterns are there to deal with this - something simple seems to get very complicated. The file I am parsing is many gigabytes so there is not enough ram to load it into ram first.
stream.on("data", f=>{
//somehow deal with a async recursive descent parser - saving the state between each chunk of data
}
Thanks for any thoughts.
Aucun commentaire:
Enregistrer un commentaire