|
By genome - Sunday, March 12, 2017
|
Hi All,I have been working for sometime on a tool to allow automatic geocoding (ie. determining latitude and longitude) of places in GenoPro data. I originally started the project using prototype GenoProX data files (json format) but then decided to try and get it to work with .gno files. Fortunately I found a useful open source javascript library to help with the unzipping and zipping of .gno files (compressed XML)  I am using Google Maps and the Google Geocoder API. The tool takes a ,gno file, unzips it and then proceeds to create markers on a Google Map for each place. Places with existing latitude and longitude values are also marked. The results can be saved into a new .gno file. Marker colours are set as follows Blue - places already geocoded, i.e. Place record already has Latitude & Longitude tags set. Green - places successfully matched and geocoded by Google Yellow - places only partially matched by Google. The location values for these places are not saved in .gno file unless confirmed by you. Light Blue - places manually located by you positioning the marker. The markers can be repositioned by dragging or removed if required. You can click on a place in the drop down list to zoom to it on the map. e.g. clicking on a place highlighted in yellow zooms to it and a dialogue allows you to confirm the position or remove it. Hover over a place in that list to see its status. Places that fail automatic geocoding are shown in red. Google imposes quotas on the number and rate of geocoding requests made. If the quota is reached then the script backs off for an increasing delay period before resuming the process. If you have more than a few hundred places then the delay can be become excessive and so I suggest if that happens you halt the process, save the results so far and resume another day. Once geocoding is complete or after halting the process you can review the results, moving or removing markers as you see fit. You can then click again for the download dialogue. There can be a delay before the download window appears so be patient. You can resume processing after download e.g. make more corrections, and then download the results again. This software is supplied 'as is' and no warranty whatsoever is given or implied. I strongly suggest that you do not download overwriting your existing data but instead choose a new filename. Load the saved .gno in GenoPro to check results. I have only tested this with Google's Chrome browser and will not be invesigating any issues that arise with other browsers. N.B. this package is NOT a GenoPro Report Generator skin. You may download and unzip it into any folder you choose. See later posts in this topic for updated zip to download. Double click the index.htm file to run the app. You can also run the app from a Web Server but in that case you will need to obtain a Google Maps JavaScript API key (see https://developers.google.com/maps/documentation/javascript/get-api-key#key)and include it in the query string of the URL used to launch the app e.g. http://www.mywebserver.org/index.htm?key=your-google-javascript-api-key-goes-here
|
|
By appleshaw - Monday, March 13, 2017
|
Downloaded and tried to run in Firefox. It spent ages unzipping but I gave up and downloaded Chrome Version 57.0.2987.98 (64-bit). No better.
Can not move initial box; no problem. However loaded the attached file and it spent ages unzipping and no result in parent directory. I used Inspect and it highlights div#map. However I do not know/understand what is going on (or not going on) so will wait for comment
|
|
By genome - Monday, March 13, 2017
|
Your .gno ran fine for me in Chrome and FireFox, unzipping in the blink of an eye.
However I usually click on the box and use the 'Open' dialogue to locate the .gno. I just noticed that if you use drag and drop then the file must be dropped on the control button/box, not on the map, otherwise a 'Save As' dialogue appears. I need to fix that. Does that help?
|
|
By appleshaw - Monday, March 13, 2017
|
Have tried again on two different PCs running Win7 and even a laptop with XP. No luck
Any thoughts? Is it the OS, download or just me?
|
|
By genome - Monday, March 13, 2017
|
No it's not you it's me and the download! There are some files missing from the download. I had forgotten I had moved some javascript libraries out to a shared folder. So I blame it on a (another) 'senior moment'. Apologies, I only discovered this when I tried downloading it on to another PC.
I have now updated the zip file in my original post in this thread and tested it by downloading to a Windows 7 PC
|
|
By appleshaw - Tuesday, March 14, 2017
|
It is now happily (I hope) running on a largeish file - 1300 places. What does 'waiting 1s' mean? Google is slow to respond?
I should have mentioned in my first post that when running the initial download I could not move the first box and the selection of area was a small box without any options. In addition on the slow XP machine it did mention a missing file, but this was not consistent and I was not sure that it was my installation was at fault. Should have mentioned this as clue to what was not happening.
Just noticed it took 4s to find Saxmundham Suffolk and 8s for Tasmania
I have always been disappointed that the Google API knows where Uxbridge is in Canada but not in England. Looking forward to seeing the final result of Geocode
|
|
By genome - Tuesday, March 14, 2017
|
|
The waiting messages indicate that a quota limit imposed by Google on geocoding requests has been reached, as indicated in my first post. I think the published limits are around 2500 per day and 25 per second but in practice I seem to get less than that. When a limit is reached, the script waits 1 second before re-attempting the request. If it is refused again the delay is doubled, this continues until the request is accepted. You can halt the process at anytime if the delay becomes to great, then you save the results in a new .gno and then continue with that updated .gno another day when you will have fewer places to geocode and a fresh daily quota.
|
|
By maru-san - Tuesday, March 14, 2017
|
|
Its a great tool but it makes me think how to organize certain families, because there are just too many places in my summary file. Another problem is there are a number of double place names due to the fact that immigrants named there new place in the USA or otherwise according to their home place at home(Germany/Denmark).
|
|
By genome - Wednesday, March 15, 2017
|
Rather than reorganize the way you record families maybe I can improve the way the geocoder works. I am going to introduce an option not to add markers for places that are already geocoded in the ,gno file. In this way if you have many many places then you will be able to geocode for example 500 places, then halt the process and save the results, then run again excluding already geocoded places and geocode the next 500 places and so on.
Double place names should not be a problem if the country is included in the place name, either directly in the name or via hierarchic places or address details. If the tool proves popular I can also provide for 'localization' so that interested users can submit translations of the texts in the app.
|
|
By genome - Thursday, March 16, 2017
|
I have now added an option not to create markers for previously geocoded places. Thus the map is less cluttered with markers and geocoding can be performed as a series of 'chunks' of places if required.
The 'app' seems to run OK in Chrome, Firefox and MS Edge but not IE. I have added code to check for required HTML5 featues and give an error message if features are not available. see later post for an updated zip file to download. Rather than me providing different language versions of the app, it is possible to install the 'Google Translate' Chrome Extension and use that to translate the app running under Chrome e.g  I have no idea whether or not Google Translate makes a reasonable attempt of translating my English text so any feedback appreciated. I notice that parts of Place names in the drop down list are also translated but makes no difference to the result.
|
|
By maru-san - Friday, March 17, 2017
|
I have a number of places, which are not recognizable by Google such as a ships name(transportation,..) and oceans. Is there a way to eliminate these from a search?
I know a ship is registered somewhere, but I did not work in that city/country. This goes as well with immigrants(arrival by ..). or on a bus(tour guide) or on a train or airplane.
|
|
By genome - Friday, March 17, 2017
|
I will add options to exclude Places with Category 'TransportationFacility' and/or 'Waters'.
The real problem is that modes of transport are not exactly places but 'things' or entities. They exist at a 'place' at a particular point in time. What is required is some way of recording a trip/journey/migration. Perhaps your earlier suggestion of sub-events could help with this, with each such sub-event marking a point/place oin a journey. Hopefully this can be implemented in GenoProX BTW, I will also be adding an 'Overlapping Marker Spiderfier' feature to help with crowded markers, clicking on such a marker will cause overlapping markers to fan out so thst each can be identified by hovering over and clicked on if required.
|
|
By genome - Saturday, March 18, 2017
|
Here is an update with 'overlapping marker spiderfier' and options to exclude places with categories Transportation Facility or Waters and as a catch all if needed, category 'Other'.
Download and unzip to any folder and open index.htm in your browser (Chrome, Firefox or Edge)
|
|
By appleshaw - Saturday, March 18, 2017
|
Am trying the today's update. Have hit a problem that in some cases it is not possible to remove the geocoding. I can drag the location to a new position and use reset to return to the original but clicking on remove coding, no confirmation menu. Can remove after this.
Have moved the problem into the North Sea
|
|
By appleshaw - Saturday, March 18, 2017
|
Another location problem - although in GenoPro but revealed using Geocoder. I am looking at unresolved locations and see the location. I can use the original data to search and the result appears in the table in Places and Locations. First problem is that this is not linked to anywhere useful. In my first case I searched a report and found the text appeared as the location on the Union tab
Now looking at another report and the search finds the site in the left of the page but it does not appear in the right hand panel.
I can obviously use a text search but hope there is an easier way
|
|
By genome - Saturday, March 18, 2017
|
re 'Remove Geocoding' issue. Just wondering if there is any error indication on the Chrome Developers Tools Console? (F12). If not, did you inadvertently click the box for 'Prevent this page displaying additional dialogs' ? If so you will need to close that tab and reopen it to recover the prompts.
Not sure what you mean regarding your 2nd location isue. Left and Right panels? is this the Narrative Report? or maybe a GenoPro 'stray object' ? (see Table Layout -> Delete -> Stray Objects)
|
|
By appleshaw - Saturday, March 18, 2017
|
Am using Firefox, so not sure what to do. F12 does show code.
The two entries I could not remove are now with green markers and show options Reset or Remove Geocoding. Clicking Remove Geocode has no effect. Reset does, which means I have 'lost' that record back to its original position. Will have to try to find it. somehow.
The panels are those of the Narrative Report. However I am now looking at the xml export to track down problems. Melbourne Australia in Scotland is easy to deal with and this showed a green icon at Hadfield, which is in Derbyshire.
Using the xml report to identify Places is not an obvious way to go, so any suggestions on how to find where a Place occurs, so it can be identified, will be useful, particularly to others not familiar with the xml structure.
Finally a question about how the classification takes place. It did not find Melbourne, Australia or Toronto, Canada but simple Hadfield was placed in Australia. I would have thought that would have been a query, so how to identify similar Problems?
Having quibbled about this there is no doubt that this an extremely useful addition to GenoPro.
One option to consider is to focus on the queries by also suppressing green icons. I find it useful to process a small number before the map gets cluttered. Dealing with over 1000 place names
|
|
By appleshaw - Saturday, March 18, 2017
|
Just moved Vancouver from the middle of Scotland to the Canadian west coast, where I found an icon for Brixton, Surrey. Thought this was stupid until I zoomed in to locate Vancouver and found a suburb of Surrey.
Then looked in the xml and the place has the following code, with new co-ordinates.
<Place ID="place00074">
<Name>Brixton, Surrey</Name>
<Country>U.K.</Country>
<Latitude>49.1120433</Latitude>
<Longitude>-122.82266930000003</Longitude>
</Place>
So the Country is there but ignored. Can it be forced to look for it?
|
|
By maru-san - Saturday, March 18, 2017
|
|
Here is another way to check the location: http://www.geoplaner.com/ if something goes wrong with google.
|
|
By appleshaw - Sunday, March 19, 2017
|
Just tried a test to see if it is possible to encourage Google to find places. Using my two state capitals it did not find I created a dummy file with Toronto, Canada; Australia and Melbourne, parent place Australia. Both ended up in the default location of the Scottish Uplands.
So I now will need to check all locations, including ones Google thinks it knows ie green markers.
To make this easier (I have a bad memory of what I have done) it would be useful if, when a green marker is Confirmed location it joined the blue group and disappeared from display.
Confirmation of yellow markers would be safer as two stage change; yellow to green and second confirmation to blue.
Finally to stop overload of the display an option to pause (Halt is the option at present) after a set number of markers (green and yellow) are displayed
|
|
By genome - Monday, March 20, 2017
|
Ok, still a few issues, but then I did say 'experimental' in my topic description  I made some changes in one of the interim releases that has caused the problems you describe. I can fix that. What my code tried to do is decide whether the place name or address ends with country name, because if it doesn't I want to use the default country code to restrict Google's search. A bug in my code means this is not working correctly. I take the last part of a place name (following the last comma, or whole name if no commas) and compare it with a list of countries. The problem was I was not stripping leading and trailing spaces before the comparison. Also when concatenating hierarchic place names, I was not adding a separating comma. The approach of comparing place names has serious limitations as my country name list is severely restricted in the languages covered (i.e. just the official language of each Country and in some cases English). I think I will have to make the default option to be not to have a default country. Another solution might be to use the internationally recognized country codes. Your place name 'Brixton, Surrey' with country U.K. is an interesting one. Brixton, Surrey is an historical address, as Brixton is now in the London Borough of Lambeth, and not the county of Surrey. Also, my code does not currently recognize 'U.K.' as a country, but UK or United Kingdom would be. But not withstanding that, there is still a problem. I use EITHER the (hierarchic) place name OR the postal address, if present, as the address to be geocoded. This did not include a mixture of the two. I have fudged such that if the only address component present is the country and the place category is not 'Country' then the Country field value is appended to the place name before geocoding. Actually if 'U.K.' is appended to the place name giving 'Brixton, Surrey, U.K.' Google still does not find the correct place, but gives the centre of Surrey. However if 'City' is set to Brixton and County/Region set to Lambeth the correct place is found. The solution to handle non-current historical place names I think is to set the postal address fields to the current address. I have attached another reincarnation of this 'app', that fixes the issues described in para 2 above and adds a 'Hide Marker' button so that after visual verification you can de-clutter the map. Also renamed Halt button to Pause and some other bits and pieces to improve messages. If I get time I will look at adding a form to allow changing of place names, address fields and categories. This will allow issues with geocoding to be resolved within the 'app' without the need to amend these details in GenoPro.
|
|
By appleshaw - Tuesday, March 21, 2017
|
Had a quick look and only one comment so far. Tried to click Index.htm and it part started but stuck on initial screen without ability to do anything (I know you said use File Open) Copied the files from the shared folder into js and seems to start properly then. Thought for final distribution.
Gardening calls so will look again later.
Point about spaces around names. I have often thought it would be useful if GenoPro removed these. I noticed in the list of names there appear to be duplicates which are caused by extra space at the end. Use of copy/paste can easily lead to these and it is not initially obvious.
Option for GenoProX?
|
|
By appleshaw - Tuesday, March 21, 2017
|
Tried a few more and only one very obvious mis-match of Middlesex, U.K. over the town of Middlesex in New York state. This had a yellow marker, so not convinced it was correct.
Seems to think that Woolwich, Kent is north of the river.
Generally near enough but need checking eg Parish Church, Croydon was identified as the Catholic church. Zooming in on Church Street did not see the old church, so just shows that it is necessary to check locations, and clearing the display after confirmation is a useful feature.
Had not realised that the map cane be zoomed with the mouse wheel. Life is always full of surprises and some of them are nice.
|
|
By genome - Wednesday, March 22, 2017
|
Thanks for testing further. I am working on the enhancement to allow editing of place name, address, category and parent.
I have also found a few of bugs that I will correct. Firstly if you hide a marker then that place's location coordinates are not saved in the .gno (as I was relying on the marker position for the values). Secondly if existing coords in the .gno are invalid e.g. not a valid number, then this can cause Google Maps API to loop and eventually crash. You will notice a slow down. If the existing coordinates are not in signed decimal format (e.g. as degrees, minutes and seconds NESW) they also cause issues. I wrote a conversion function for this but forgot to use it! Sometimes clicking the box for the open file window does nothing. I believe that Google Maps is hanging on to the mouse click event and my code doesn't see it. Unsure as to the solution to this at present but often clicking the map first and then the box works, and sometimes multiple clicks on the box kick starts it too.
|
|
By appleshaw - Wednesday, March 22, 2017
|
If the hidden data are not saved I will await next version.
It has proved a useful error spotter. Grimsby, Turkey? This had me puzzled as editing of place Grimsby showed no parent place. Took me some time to spot Turkey had been added as Country. Don't ask me how or why. Only link is that he had fought at Gallipoli but the record related to his death.
|
|
By genome - Friday, March 31, 2017
|
Well it has been a while but I think the next iteration is ready. I have added code to allow changes to place name, address, category and parent place. This gives the ability to adjust details so that Google gets the correct location rather than having to manually move the marker to its correct place. Hidden markers for new/changed geocodings are now saved if .gno downloaded.
It hasn't been exhaustively tested but at present I am exhausted from testing! So the usual caveats apply - don't overwrite your precious .gno, save to another file and check results please. This software is offered 'as is' without any warranty or guarantee, users do so at their own risk. If you find any problems or have further suggestions then respond here.
|
|
By appleshaw - Saturday, April 1, 2017
|
Tried to edit a place (first in file) but could not get out of edit menu. Tried adding some data but neither the cancel or Ok moved on.
Nothing to do with Geocoder but have you any thoughts on tracking down, within GenoPro, the link between Place and the person containing the reference? At present I am using a text editor to search the xml file but suspect that I have overlooked the obvious.
The need for this is that many place names I have used are copy/pasted from census etc and in many cases it is a street address without town. (This could be another item for Problem Spotter, although it would not be simple, although query where address does not have a parent would be a start - as long as it could be excluded from check)
Back to gardening
|
|
By genome - Saturday, April 1, 2017
|
Most of my testing is with Chrome and it appears that FireFox handles events differently, so at the moment those buttons only work in Chrome.
The attached version fixes it for Firefox.Yes, you could be overlooking the obvious when it comes to finding references to places. Just use the References tab of the Place Properties dialogue.
|