<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0mm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0mm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Calibri","sans-serif";}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#44546A;}
span.EmailStyle20
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#993366;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#003300;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-GB link="#0563C1" vlink="#954F72"><div class=WordSection1><p class=MsoNormal><span style='color:#003300'>Actually it’s not as bad as that.<o:p></o:p></span></p><p class=MsoNormal><span style='color:#003300'>The QMs are all parsed when troggle imports the svx files. But Troggle just doesn’t have an *<b>output page</b>* to show them. <o:p></o:p></span></p><p class=MsoNormal><span style='color:#003300'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#003300'>That should be a relatively easy addition.<o:p></o:p></span></p><p class=MsoNormal><span style='color:#003300'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0mm 0mm 0mm'><p class=MsoNormal><b><span lang=EN-US style='font-size:10.0pt;font-family:"Tahoma","sans-serif";mso-fareast-language:EN-GB'>From:</span></b><span lang=EN-US style='font-size:10.0pt;font-family:"Tahoma","sans-serif";mso-fareast-language:EN-GB'> Philip Sargent (Gmail) [mailto:philip.sargent@gmail.com] <br><b>Sent:</b> 16 April 2020 21:56<br><b>To:</b> 'Rob Watson'<br><b>Subject:</b> FW: QMs & svx2qm.py<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='color:#993366'>Rob,<o:p></o:p></span></p><p class=MsoNormal><span style='color:#993366'>since you did the documentation for QMs, you might be interested in quite how broken the website/troggle system is for QMs.<o:p></o:p></span></p><p class=MsoNormal><span style='color:#993366'>We have a moderately clear architecture, but all the input procedures and parsers are all over the place.<o:p></o:p></span></p><p class=MsoNormal><span style='color:#993366'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#993366'>At the moment it looks like the command line tool </span><b>svx2qm.py </b><span style='color:#993366'>does a good job in extracting all the QMs from all the svx files. So I will document that now.<o:p></o:p></span></p><p class=MsoNormal><span style='color:#993366'>This is completely independent of the website.<o:p></o:p></span></p><p class=MsoNormal><span style='color:#993366'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#993366'>By next serious expo in 2021 I hope this part of the system is re-written.<o:p></o:p></span></p><p class=MsoNormal><span style='color:#993366'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#993366'>Philip</span><b><o:p></o:p></b></p><p class=MsoNormal><span style='color:#993366'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#993366'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0mm 0mm 0mm'><p class=MsoNormal><b><span lang=EN-US style='font-size:10.0pt;font-family:"Tahoma","sans-serif";mso-fareast-language:EN-GB'>From:</span></b><span lang=EN-US style='font-size:10.0pt;font-family:"Tahoma","sans-serif";mso-fareast-language:EN-GB'> Philip Sargent (Gmail) [mailto:philip.sargent@gmail.com] <br><b>Sent:</b> 16 April 2020 21:43<br><b>To:</b> philip@tecnocode.co.uk; expo-tech@lists.wookware.org<br><b>Subject:</b> QMs & svx2qm.py<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='color:#44546A'>It looks like the files </span><b>svx2qm.py</b> <span style='color:#44546A'>doesn’t produce the files troggle imports, but some perl scripts do.<o:p></o:p></span></p><p class=MsoNormal><span style='color:#44546A'>Which is even worse.<o:p></o:p></span></p><p class=MsoNormal><span style='color:#44546A'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#44546A'>So it looks to me that Phil Withnall’s </span><b>svx2qm.py</b><span style='color:#44546A'> code should be repurposed into a “proper” qm-import module which<o:p></o:p></span></p><p class=MsoNormal><span style='color:#44546A'>would eventually live in troggle/parsers/ <o:p></o:p></span></p><p class=MsoNormal><span style='color:#44546A'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#44546A'>The perl and CSV stuff are in :expoweb: not :loser: repos.<o:p></o:p></span></p><p class=MsoNormal><span style='color:#44546A'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#44546A'>troggle/parsers/qms.py currently imports into troggle using a mixture of csv and html parsers:<o:p></o:p></span></p><p class=MsoNormal><span style='color:#44546A'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#44546A'>parseCaveQMs(cave='stein',inputFile=r"1623/204/qm.csv")<o:p></o:p></span></p><p class=MsoNormal><span style='color:#44546A'>parseCaveQMs(cave='hauch',inputFile=r"1623/234/qm.csv")<o:p></o:p></span></p><p class=MsoNormal><span style='color:#44546A'>parseCaveQMs(cave='kh', inputFile="1623/161/qmtodo.htm")<o:p></o:p></span></p><p class=MsoNormal><span style='color:#44546A'>#parseCaveQMs(cave='balkonhoehle',inputFile=r"1623/264/qm.csv")<o:p></o:p></span></p><p class=MsoNormal><span style='color:#44546A'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#44546A'>And possibly these .csv files are produced by <o:p></o:p></span></p><p class=MsoNormal><span style='color:#44546A'>qmreader.pl<o:p></o:p></span></p><p class=MsoNormal><span style='color:#44546A'>or possibly<o:p></o:p></span></p><p class=MsoNormal><span style='color:#44546A'>tablize-qms.pl<o:p></o:p></span></p><p class=MsoNormal><span style='color:#44546A'>which are all in expoweb/1623/204/.<o:p></o:p></span></p><p class=MsoNormal><span style='color:#44546A'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#44546A'>1623/204/qms.html is definitely produced by<o:p></o:p></span></p><p class=MsoNormal><span style='color:#44546A'>tablize-qms.pl<o:p></o:p></span></p><p class=MsoNormal><span style='color:#44546A'>from<o:p></o:p></span></p><p class=MsoNormal><span style='color:#44546A'>qms.csv<o:p></o:p></span></p><p class=MsoNormal><span style='color:#44546A'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#44546A'>in 1623/234<o:p></o:p></span></p><p class=MsoNormal><span style='color:#44546A'>there is just tablize-qms.pl , qm.csv and qm.html<o:p></o:p></span></p><p class=MsoNormal><span style='color:#44546A'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#44546A'>in 1623/161 there is the qmtodo.htm and a qmdone.htm but no csv file<o:p></o:p></span></p><p class=MsoNormal><span style='color:#44546A'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#44546A'>in 1523/264 there is a qm.csv which apparently was hand-edited?<o:p></o:p></span></p><p class=MsoNormal><span style='color:#44546A'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#44546A'>It all needs someone to properly take it in hand and sort it all out.<o:p></o:p></span></p><p class=MsoNormal><span style='color:#44546A'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#44546A'>Note that this all predates Tunnocks; and 'balkonhoehle' is commented out. Nothing at all for FGH, GSM, HC etc.<o:p></o:p></span></p><p class=MsoNormal><span style='color:#44546A'>Philip<o:p></o:p></span></p><p class=MsoNormal><span style='color:#44546A'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0mm 0mm 0mm'><p class=MsoNormal><b><span lang=EN-US style='font-size:10.0pt;font-family:"Tahoma","sans-serif";mso-fareast-language:EN-GB'>From:</span></b><span lang=EN-US style='font-size:10.0pt;font-family:"Tahoma","sans-serif";mso-fareast-language:EN-GB'> Philip Withnall [mailto:philip@tecnocode.co.uk] <br><b>Sent:</b> 13 April 2020 23:41<br><b>To:</b> Philip Sargent (Gmail)<br><b>Subject:</b> Re: svx2qm<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif";mso-fareast-language:EN-GB'>Hi Philip,<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif";mso-fareast-language:EN-GB'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif";mso-fareast-language:EN-GB'>Hope you’re well, thanks for getting in touch about this.<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif";mso-fareast-language:EN-GB'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif";mso-fareast-language:EN-GB'>On Sat, 2020-04-11 at 20:28 +0100, Philip Sargent (Gmail) wrote:<o:p></o:p></span></p></div><blockquote style='border:none;border-left:solid #729FCF 1.5pt;padding:0mm 0mm 0mm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0mm;margin-bottom:5.0pt'><p class=MsoPlainText>Hi,<o:p></o:p></p><p class=MsoPlainText>I am trying to document the non-troggle scripts used by the expo website.<o:p></o:p></p><p class=MsoPlainText><a href="http://expo.survex.com/handbook/troggle/otherscripts.html">http://expo.survex.com/handbook/troggle/otherscripts.html</a> <o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>I see that your <b>svx2qm.py</b> lives in the :loser: repo in /qms/<o:p></o:p></p><p class=MsoPlainText>But that means that the files generated (csv or human) are not served by the webserver.<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p></blockquote><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif";mso-fareast-language:EN-GB'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif";mso-fareast-language:EN-GB'>The generated files are not meant to be served by the webserver, it’s a tool for people to run locally. Someone could modify it to create HTML output (or post-process the CSV output to do the same), but that is work still to be done.<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif";mso-fareast-language:EN-GB'><o:p> </o:p></span></p></div><blockquote style='border:none;border-left:solid #729FCF 1.5pt;padding:0mm 0mm 0mm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0mm;margin-bottom:5.0pt'><p class=MsoPlainText>I can't see any problem with moving it all to <span style='font-family:Consolas'>expoweb/scripts/</span> - so long as it is run with the loser top level directory specified - but I might be mistaken:<o:p></o:p></p><p class=MsoPlainText><span style='font-family:Consolas'>find /home/expo/loser -name '*.svx' | xargs ./svx2qm.py --format human<o:p></o:p></span></p><p class=MsoPlainText>and it should go into the Makefile too at some point.<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p></blockquote><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif";mso-fareast-language:EN-GB'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif";mso-fareast-language:EN-GB'>Feel free to move it wherever; I am not planning on doing any further work on it. The script itself just expects to be passed some (relative or absolute) paths to SVX files, so can be placed wherever, as long as it’s passed appropriate relative paths.<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif";mso-fareast-language:EN-GB'><o:p> </o:p></span></p></div><blockquote style='border:none;border-left:solid #729FCF 1.5pt;padding:0mm 0mm 0mm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0mm;margin-bottom:5.0pt'><p class=MsoPlainText><b>How and who usually uses the output?</b> Is there another script somewhere producing an HTML page ?<o:p></o:p></p><p class=MsoPlainText>Or should I do something like this<o:p></o:p></p><p class=MsoPlainText><a href="http://expo.survex.com/xmlvalid.html">http://expo.survex.com/xmlvalid.html</a> <o:p></o:p></p><p class=MsoPlainText>or<o:p></o:p></p><p class=MsoPlainText><a href="http://expo.survex.com/svxvalid.html">http://expo.survex.com/svxvalid.html</a> <o:p></o:p></p><p class=MsoPlainText>to display the text output ?<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>Or maybe even edit it into a troggle "view" so that it become an integral part of troggle ?<o:p></o:p></p></blockquote><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif";mso-fareast-language:EN-GB'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif";mso-fareast-language:EN-GB'>I haven’t written any other scripts which post-process the data or otherwise format it.<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif";mso-fareast-language:EN-GB'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif";mso-fareast-language:EN-GB'>I guess it all depends on what questions people are trying to answer using the QM data, as to how (and where) best to present it. I’m afraid I don’t have any suggestions there.<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif";mso-fareast-language:EN-GB'><o:p> </o:p></span></p></div><blockquote style='border:none;border-left:solid #729FCF 1.5pt;padding:0mm 0mm 0mm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0mm;margin-bottom:5.0pt'><p class=MsoPlainText>Rob Watson wrote some documentation about QMs<o:p></o:p></p><p class=MsoPlainText><a href="http://expo.survex.com/handbook/survey/qmentry.html">http://expo.survex.com/handbook/survey/qmentry.html</a> <o:p></o:p></p><p class=MsoPlainText>is there anything subtle missing as to how they are used ?<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p></blockquote><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif";mso-fareast-language:EN-GB'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif";mso-fareast-language:EN-GB'>Nope, I think Rob’s page covers it all. That page also documents the correct QM format which is what svx2qm.py understands. (There were some older or artisanal QM formats floating around at one point, although I think I reformatted them all so the tool would understand them, and so people would hopefully standardise on what Rob’s documented from then on.)<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif";mso-fareast-language:EN-GB'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif";mso-fareast-language:EN-GB'>Philip<o:p></o:p></span></p></div></div></body></html>