Full XML Parser for Wikis

You may want a primer on parsing XML with Ada. That is an... okay... example, but better than many Ada packages have on the web. The following is a more complete, but more complicated example. In XML/Ada you supply an object to the parser that provides the rules for how to parse the document. It is a pretty easy to work with and extend. This code was some of the earliest I started writing in this whole adventure. I waited to finalize it until now so I could completely figure out how the client code would work with it.

This is the client to the XML/Ada library and makes use of the parser object posted below: (test_wiki_parser.adb)
with Ada.Command_Line; with Ada.Text_IO; with Wiki_Readers; with Input_Sources.Large_File; with Sax.Readers; with Ada.Strings.Unbounded; procedure Test_Wiki_Parser is File_Source : Input_Sources.Large_File.Large_File_Input(16); Wiki_Parser : Wiki_Readers.Wiki_Reader; -- passed to the Wiki_Parser to put the title to standard out procedure Put_Title (Article : in Wiki_Readers.Full_Document) is begin Ada.Text_IO.Put_Line(Ada.Strings.Unbounded.To_String(Article.Title) & " is article # " & Integer'Image(Article.Number)); end Put_Title; -- passed to the XML_Parser to relay the # of documents in the collection procedure Put_Num_Articles (Value : in Natural) is begin Ada.Text_IO.Put_Line(Integer'Image(Value) & " documents in collection"); end Put_Num_Articles; begin if Ada.Command_Line.Argument_Count /= 1 then Ada.Text_IO.Put_Line ("Need to specify a filename!"); return; end if; Wiki_Readers.Set_Processes (Handler => Wiki_Parser, Document_Process => Put_Title'Access, Collection_Process => Put_Num_Articles'Access); Input_Sources.Large_File.Open(Filename => Ada.Command_Line.Argument(1), Input => File_Source); Wiki_Readers.Parse (Parser => Wiki_Parser, Input => File_Source); Input_Sources.Large_File.Close (File_Source); exception when Sax.Readers.XML_Fatal_Error => Input_Sources.Large_File.Close (File_Source); end Test_Wiki_Parser;
This uses the Input_Sources.Large_File posted earlier. This entire program boils down to:

• Tell parser what to do
• Open file
• Parse file
• Close file

(wiki_reader.ads)
with Sax.Readers; with Sax.Attributes; with Unicode.CES; with Ada.Strings.Unbounded; --use Ada.Strings.Unbounded; package Wiki_Readers is type Wiki_Reader is new Sax.Readers.Reader with private; -- Stores information on documents type Full_Document is record Number : Natural := 0; -- What article # this is Title : Ada.Strings.Unbounded.Unbounded_String; Text : Ada.Strings.Unbounded.Unbounded_String; end record; procedure End_Document (Handler : in out Wiki_Reader); procedure Start_Element (Handler : in out Wiki_Reader; Namespace_URI : Unicode.CES.Byte_Sequence := ""; Local_Name : Unicode.CES.Byte_Sequence := ""; Qname : Unicode.CES.Byte_Sequence := ""; Atts : Sax.Attributes.Attributes'Class); procedure End_Element (Handler : in out Wiki_Reader; Namespace_URI : Unicode.CES.Byte_Sequence := ""; Local_Name : Unicode.CES.Byte_Sequence := ""; Qname : Unicode.CES.Byte_Sequence := ""); procedure Characters (Handler : in out Wiki_Reader; Sequence : Unicode.CES.Byte_Sequence); -- Takes in pointers to procedures on what to do at end of each -- document and at the end of the collection procedure Set_Processes (Handler : in out Wiki_Reader; Document_Process : access procedure (Article : Full_Document); Collection_Process : access procedure (Value : Natural)); private -- For indicating what field the characters come from. type Character_Field_Type is (Unknown, Text, Title); type Wiki_Reader is new Sax.Readers.Reader with record Document_Process : access procedure (Article : Full_Document); Collection_Process : access procedure (Value : Natural); Character_Field : Character_Field_Type := Unknown; Valid_Title : Boolean; -- Validity of current title (not image, category, or other) Page : Full_Document; end record; end Wiki_Readers;
(wiki_reader.adb)
with Sax.Readers; use Sax.Readers; with Unicode.CES; use Unicode.CES; with Unicode; use Unicode; with Sax.Attributes; with Ada.Strings; package body Wiki_Readers is ------------------ -- End_Document -- ------------------ procedure End_Document (Handler : in out Wiki_Reader) is begin Handler.Collection_Process(Handler.Page.Number); end End_Document; ------------------- -- Start_Element -- ------------------- procedure Start_Element (Handler : in out Wiki_Reader; Namespace_URI : Unicode.CES.Byte_Sequence := ""; Local_Name : Unicode.CES.Byte_Sequence := ""; Qname : Unicode.CES.Byte_Sequence := ""; Atts : Sax.Attributes.Attributes'Class) is begin -- Only care about start of title and text if Local_Name = "title" then Handler.Character_Field := Title; -- Opening tag of title elsif Local_Name = "text" and Handler.Valid_Title then Handler.Character_Field := Text; -- Opening tag of text end if; end Start_Element; ----------------- -- End_Element -- ----------------- procedure End_Element (Handler : in out Wiki_Reader; Namespace_URI : Unicode.CES.Byte_Sequence := ""; Local_Name : Unicode.CES.Byte_Sequence := ""; Qname : Unicode.CES.Byte_Sequence := "") is begin Handler.Character_Field := Unknown; if Local_Name = "page" then -- at the end of a page begin if Handler.Valid_Title and Ada.Strings.Unbounded.Element(Handler.Page.Text, 1) /= '#' then -- title is valid, and article is not blank or a redirect, so the article is passed on Handler.Page.Number := Handler.Page.Number + 1; Handler.Document_Process(Handler.Page); end if; exception when ADA.STRINGS.INDEX_ERROR => null; end; -- unconditional re-initialization Handler.Page.Title := Ada.Strings.Unbounded.Null_Unbounded_String; Handler.Page.Text := Ada.Strings.Unbounded.Null_Unbounded_String; elsif Local_Name = "title" then -- Done parsing the characters for the title, check if it is valid or not Handler.Valid_Title := Ada.Strings.Unbounded.Index(Handler.Page.Title, ":") = 0; end if; end End_Element; ---------------- -- Characters -- ---------------- procedure Characters (Handler : in out Wiki_Reader; Sequence : Unicode.CES.Byte_Sequence) is begin if Handler.Character_Field = Text then Ada.Strings.Unbounded.Append(Handler.Page.Text, Sequence); elsif Handler.Character_Field = Title then Ada.Strings.Unbounded.Append(Handler.Page.Title, Sequence); end if; end Characters; procedure Set_Processes (Handler : in out Wiki_Reader; Document_Process : access procedure (Article : Full_Document); Collection_Process : access procedure (Value : Natural)) is begin Handler.Document_Process := Document_Process; Handler.Collection_Process := Collection_Process; end Set_Processes; end Wiki_Readers;
I tried passing the two processes (Document_Process & Collection_Process) at the declaration of Wiki_Parser, but evidently Ada does not like access types being passed there (and a compiler bug). I could possibly wrap them in a record and pass them through that also. I think this way is not the preferred way, but it works.

Yay closures.

This combo fully parses the new 14 GB version of Wikipedia in 25 minutes. This is not the exact version that will be used in the final product. The Full_Document type will be declared elsewhere. Other than that all the code above will be included.