Old Weather Forum

Old Weather: Arctic => Help => Topic started by: Bob on 19 February 2017, 15:06:54

Title: Project Updates
Post by: Bob on 19 February 2017, 15:06:54
Project Update

The next step for Old Weather transcribing is almost ready for wide release. We're still tweaking some things and will be doing a full test run on one of the Bear log sets soon to work the bugs out of the process. In the meantime, here is an introduction to what we have. Let's call it OW:Next for now.

What It Is

OW:Next is a system for connecting transcribers with the logs, providing a software tool for transcribing the pages, and a means for transferring the data and tracking the work. The transcribing tool is a LibreOffice Calc spreadsheet, specially formatted for each log set, with automated features for working with the log page images. OW:Next keeps the existing forum as the central place for interacting with the project. Creating it, maintaining it and operating it are in the hands of the participants.

What It Is Not

OW:Next is not a one-stop web interface like previous iterations of the Old Weather project. It does not (yet) fully automate all the steps in the transcribing process. While we've tried to make this new system as accessible as possible, it might not be everyone's cup of tea at this point.

Getting Involved

If you're ready to roll up your sleeves a bit, install some software (free), and get down to more of a nuts-and-bolts type of transcribing experience, then let us know. You can post here on the forum or send us (the Moderators) a PM. Knowing the interest level will help us plan the wider roll-out.

Getting Started

We have a series of How-To presentations:

Installation and Setup (https://docs.google.com/presentation/d/1p29DrIPa5s1TZGXZwtcHOpFZ0AhFdHBLmKCgJQ7REmw/pub?start=false)
Using the Spreadsheet (https://docs.google.com/presentation/d/1F2UpcRtCIzknKtaP7GJod2CviuG6ZokuK0t9qSzNkP8/pub?start=false)
Transcribing Hints, Tips and Tricks (https://docs.google.com/presentation/d/1kMYTNU8PcIb_zjC4KeB81IXkLBT9E9WGEf7useXafNY/pub?start=false)
Uploading the Data (https://docs.google.com/presentation/d/11VRcFO4D7Sqsdhoq7nL-HjQNlKJu22tdK4M1xSiuxvs/pub?start=false)

You can have a look at these to get a feel for how things work. We're in the process of putting together a simple centralized site with basic introductory information and links. It is currently hosted here (https://sites.google.com/view/owfaq/getting-started), and although the content is a work in progress, it gives you an idea of the project organization.

Stay tuned for more news. As always, you can ask any questions or post comments here in the forum!
Title: Re: Project Updates
Post by: Bob on 10 March 2017, 08:05:26
A quick status update :

We've received detailed feedback from the science team after they've had a chance to work with the beta-test and dry-run data that's been accumulated so far. The general conclusion is that the system should work well, but there are some details that need to be attended to, mostly related to things like events tagging and data format. We'll be getting the recommended changes in place and tested, followed by another round of science team review. After that, we could be ready for a wider roll-out.  ;)

Thanks to everyone that's helped with the testing so far. We'll be in a bit of a hiatus while the updates are made, but we'll post here when we're ready for the next dry run.

Stay tuned!
Title: Re: Project Updates
Post by: Craig on 10 March 2017, 13:03:54
That's encouraging news, Bob. Thank you for all you have done with this new capture system.  8) 8) 8)
Title: Re: Project Updates
Post by: Randi on 10 March 2017, 13:13:02
Indeed!
Title: Re: Project Updates
Post by: AvastMH on 10 March 2017, 14:22:37
Ready and waiting Bob  :D
Title: Re: Project Updates
Post by: Kevin on 17 March 2017, 19:16:09
The Old Weather home page now has a 'Resources' button. I've put our imager manual there, but can put more important links to other things above that too. Can also change the name to 'push this button!'
Title: Re: Project Updates
Post by: Michael on 17 March 2017, 20:49:42
Nice!
Title: Re: Project Updates
Post by: Bob on 20 December 2017, 21:20:34
Now that the current set of OldWeather ships is essentially complete, we're pleased to announce that the next generation of OldWeather is ready to use.
So, head over to the Introduction (http://forum.oldweather.org/index.php?topic=4690.msg143598#msg143598) and read all about it.

You will find that the new system is quite a bit different from the current web-based version.
It has been developed as a set of tools for transcribing the log pages and will require more interaction from users.
So it's very important to review the introductory materials to get familiar with the overall process.
As always, you can Ask Questions Here (http://forum.oldweather.org/index.php?topic=4826.msg148448#msg148448).

Another big difference is that, for the time being, the focus is on ice observations and ship's position details.
You will find more on that in Transcribing: Ice (http://forum.oldweather.org/index.php?topic=4690.msg144397#msg144397).
You're certainly free to transcribe weather and events if that interests you, but the Science Team is asking that ice be the priority.

This system is literally a work in progress (for about a year now), and will always be one.
Many of you have helped with development inputs, beta testing, advice and suggestions.
We very much want to continue this as a user-supported platform that will evolve over time as we gain experience and take on new log sets and objectives.
So, please feel free to contribute your ideas in the forum as you work through the new system.

Also, remember we are all volunteers here, so our resources are somewhat limited, and your patience will be very much appreciated.
With that in mind, we're limiting the number of ships available at first while we weather the first round of public participation.
For now, that's the Bear (http://forum.oldweather.org/index.php?board=48.0).
Title: Re: Project Updates
Post by: Randi on 21 December 2017, 10:07:01
Currently we are doing the Pacific. After that, it will be the Atlantic.
We have Burton Island (http://forum.oldweather.org/index.php?board=50.0), Storis (http://forum.oldweather.org/index.php?board=51.0), and Staten Island (http://forum.oldweather.org/index.php?board=53.0).
Title: Re: Project Updates
Post by: Randi on 11 January 2018, 11:56:18
It admittedly is a steep learning curve, but the more I use it the better I like it!


Old Weather of the transcribers, by the transcribers, for the transcribers, shall not perish from the earth
 ;)
Title: Re: Project Updates
Post by: Hanibal94 on 11 January 2018, 14:55:52
Quote from: Randi on 11 January 2018, 11:56:18
It admittedly is a steep learning curve, but the more I use it the better I like it!

I agree completely!

Quote from: Randi on 11 January 2018, 11:56:18
Old Weather of the transcribers, by the transcribers, for the transcribers, shall not perish from the earth
 ;)

HUZZAH!
Title: Re: Project Updates
Post by: Bob on 13 January 2018, 14:32:16
Beta Testing Needed

New spreadsheets are here:

https://drive.google.com/drive/folders/0B79AVnuaPynAcjdoZG1ELWdvT2c


The new features are -

User-settable Location Wizard default startup screen position

The settings are in Cells C48 and C49 of the Setup tab. The horizontal and vertical values specify the default top left corner placement of the wizard window relative to the Calc window, in pixels. Negative values are allowed, and out of range values will just put the wizard window into one of the screen corners. Try it and see.  ;)

Added entries to Location Wizard Time drop list

New: 0800, 1200 and 2000

Fixed decimal minutes bug in Location Wizard

Maximum allowable value was set to 59 for minutes and seconds, it's now 60.

Auto-prompt for Clear Data on image load

When Get Image is called it checks for any non-empty cells in the data, event and date input ranges, and calls Clear Data if any of the ranges are not empty.

Write synoptic position entries to WR Data tab

Location Wizard writes the XML-formatted position strings to the WR Data tab for '8 AM', 'Noon' and '8 PM' times ('0800', '1200' and '2000' also trigger this).


I need someone to independently verify that these things are working in all five files.

Thanks!
Title: Re: Project Updates
Post by: Randi on 13 January 2018, 19:16:43
Storis:
User-settable Location Wizard default startup screen position OK
Added entries to Location Wizard Time drop list OK
Fixed decimal minutes bug in Location Wizard OK
Auto-prompt for Clear Data on image load OK
Write synoptic position entries to WR Data tab OK


Staten Island 1952-June1955:
User-settable Location Wizard default startup screen position OK
Added entries to Location Wizard Time drop list OK
Fixed decimal minutes bug in Location Wizard OK
Auto-prompt for Clear Data on image load OK
Write synoptic position entries to WR Data tab OK


Staten Island July-Dec1955:
User-settable Location Wizard default startup screen position OK
Added entries to Location Wizard Time drop list OK
Fixed decimal minutes bug in Location Wizard OK
Auto-prompt for Clear Data on image load OK
Write synoptic position entries to WR Data tab OK
Title: Re: Project Updates
Post by: Pommy Stuart on 13 January 2018, 21:17:22
Burton
User-settable Location Wizard default startup screen position OK
Added entries to Location Wizard Time drop list OK
Fixed decimal minutes bug in Location Wizard OK
Auto-prompt for Clear Data on image load OK
Write synoptic position entries to WR Data tab *****  not sure what you mean by this ***
Title: Re: Project Updates
Post by: Randi on 13 January 2018, 22:50:15
(https://i.imgur.com/twgqXm1.png)
Title: Re: Project Updates
Post by: Pommy Stuart on 13 January 2018, 23:51:37
Thanks. Never looked that far into the SS.
Thought if I don't need it don't look in case I do something to it.

Yes it works. OK

Q. Do I need to click 'Not specified' when ALL entries in that type of log book are 'Not Specified' as either Obs or DR??

Any idea when next ships come on line?
Title: Re: Project Updates
Post by: Randi on 14 January 2018, 00:27:01
Quote from: Pommy Stuart on 13 January 2018, 23:51:37
Q. Do I need to click 'Not specified' when ALL entries in that type of log book are 'Not Specified' as either Obs or DR??
For each lat/long entry you need to select Observed, DR, or Not Specified.
Title: Re: Project Updates
Post by: Randi on 14 January 2018, 00:42:22
With the Clear Data on image load, Hanibal's (http://forum.oldweather.org/index.php?topic=4336.msg148929#msg148929) and Joke Slayer's (http://forum.oldweather.org/index.php?topic=4646.msg149266#msg149266) data saving tricks don't work :'(
Possibly if the Clear Data was before the image load?
Title: Re: Project Updates
Post by: Hanibal94 on 14 January 2018, 04:24:04
Or maybe just exempt the date from "Clear Data"?
Title: Re: Project Updates
Post by: Bob on 14 January 2018, 08:49:14
Hmm...

Quote from: Hanibal94 on 14 January 2018, 04:24:04
Or maybe just exempt the date from "Clear Data"?
Title: Re: Project Updates
Post by: Randi on 14 January 2018, 13:08:11
Quote from: Pommy Stuart on 13 January 2018, 23:51:37
Any idea when next ships come on line?

Soon!
We're going to add Atlantic years for Staten Island and Storis.
Title: Re: Project Updates
Post by: Bob on 21 January 2018, 10:18:59
Hi, All -

I'm doing a bit of web site reorganization, and the direct links for ship's pages are changing. You can use the main ships page to find your way back to them:

https://sites.google.com/view/owfaq/ships

Under the new structure the individual ship's page links shouldn't need to change again.

Thanks, and apologies for any inconvenience!
Title: Re: Project Updates
Post by: Hanibal94 on 21 January 2018, 11:33:29
I like it - Alphabetical sorting is much better than sorting by area of activity!
Thanks!
Title: Re: Project Updates
Post by: Randi on 21 January 2018, 17:11:50
The Storis links have been updated and all years - except June and July 1944 - are available.
Title: Re: Project Updates
Post by: Bob on 22 January 2018, 19:46:59
New spreadsheets have been posted: Look for version *_02.ods

> Fixes a bug in the image number spinner that prevented it from incrementing past 500 (the limit is now 999).

> Includes the new features described in the last beta testing request. Note: 'Clear Data' is called automatically on image load if you have anything in any of the data areas. It still asks for confirmation, which you can decline if you want to keep the old data.

> The Storis spreadsheet has the URL index updated to include all years 1942-1955. Not for use with June-July, 1944; that will need a different spreadsheet [TBD].

Reminder: You'll need to enter your local directories in the Setup tab when you start with a new version.
Title: Re: Project Updates
Post by: Bob on 26 January 2018, 14:02:39
New spreadsheets have been posted: Look for version *_04.ods

> Adds 0800 and 2000 as single entries to the Event Input Wizard time drop-down list. (User request)

> Corrected a typo in the Event Input Wizard time drop-down list.

Reminder: You'll need to enter your local directories in the Setup tab when you start with a new version.
Title: Re: Project Updates
Post by: Bob on 11 February 2018, 17:28:00
New spreadsheets have been posted: Look for version *_05.ods

> Fixes a bug where blank Lat/Lon entries in the Location Wizard would be interpreted as zeros.

Reminder: You'll need to enter your local directories in the Setup tab when you start with a new version.
Title: Re: Project Updates
Post by: Bob on 12 February 2018, 19:37:26
New Eastwind spreadsheet posted (*_02_05.ods)

> Fixes incorrect column setup for distance traveled.
Title: Re: Project Updates
Post by: Pommy Stuart on 12 February 2018, 19:47:45
Thanks for Eastwind fix.
Title: Re: Project Updates
Post by: Bob on 21 February 2018, 20:48:01
New spreadsheets have been posted: Look for version *_06.ods

> Adds a warning dialog if the Date cell doesn't contain ten characters; i.e. invalid or missing date. This will be expanded in the future to do more in-depth error checking for ten-character entries.

> Expands the Clear Data confirmation message to include a 'possibly unsaved data' warning if there hasn't been a Write Data since the last Clear.

Reminder: You'll need to enter your local directories in the Setup tab when you start with a new version.
Title: Re: Project Updates
Post by: Randi on 21 February 2018, 21:52:03
8)
Title: Re: Project Updates
Post by: Michael on 22 February 2018, 10:38:14
 :) :) :)
Title: Re: Project Updates
Post by: Pommy Stuart on 22 February 2018, 16:11:59
Hi Bob.
Eastwind log page seems to have shifted a long way left and now the naughty miles column is under the LHS buttons making it unusable.
What used to be a white border in column 1 has disappeared moving the image to the left.
Widening column A does not seem to work.
I am continuing using v5 for the moment which is still OK with the same pages.
Stuart
Title: Re: Project Updates
Post by: Bob on 22 February 2018, 19:24:26
Hi, Stuart -

My fault, updated version (still 06) posted.

Eastwind is a bit quirky, it doesn't have as much left margin in the images as the other logs. The only way to get it outside the buttons is to load it into Cell B1 instead of A1. I'll probably forget that the next time, too.  ::)

Thanks,

- Bob
Title: Re: Project Updates
Post by: Pommy Stuart on 22 February 2018, 19:40:23
All good, Thanks.  ;)
Title: Re: Project Updates
Post by: Bob on 08 April 2018, 17:39:25
New spreadsheets have been posted: Look for version *_07.ods

> Rewrites the AM and PM Hour values when Clear Data is called. This corrects for when Hour values are modified, either deliberately for non-standard or additional entries, or accidentally, and prevents the changes from propagating to subsequent pages if it's not caught and corrected by the transcriber.

Reminder: You'll need to enter your local directories in the Setup tab when you start with a new version.
Title: Re: Project Updates
Post by: Randi on 08 April 2018, 17:43:47
Quote from: Bob on 08 April 2018, 17:39:25
> Rewrites the AM and PM Hour values when Clear Data is called. This corrects for when Hour values are modified, either deliberately for non-standard or additional entries, or accidentally, and prevents the changes from propagating to subsequent pages if it's not caught and corrected by the transcriber.
Excellent!
Title: Re: Project Updates
Post by: Bob on 08 April 2018, 19:00:36
Michael's idea...  ;) 8)

Quote from: Randi on 08 April 2018, 17:43:47
Excellent!
Title: Re: Project Updates
Post by: Randi on 08 April 2018, 20:34:11
Excellent Michael! ;D
Title: Re: Project Updates
Post by: AvastMH on 08 April 2018, 20:35:43
Sounds very good to me!  :D
Title: Re: Project Updates
Post by: Michael on 08 April 2018, 22:48:55
 
Quote from: Randi on 08 April 2018, 20:34:11
Excellent Michael! ;D

 :-[
Title: Re: Project Updates
Post by: Randi on 18 July 2018, 18:06:56
The science team decided to go with a single weather transcriber.
That is quite a tribute to OW transcribers!

Michael says he has processed about 135,000 hours of data and found that there have been very very few transcription errors. He notes two reasons:
1. Bob's spreadsheet entry method makes the transcriptions nearly error free.
2. Our transcribers are very dedicated. In this phase we don't have people who do just a couple of pages and quit.

Philip agrees. He says that if the transcription was being done by students or a commercial company, at least two transcribers would be needed. However, with "oldWeather experts" one should be sufficient.

Under these circumstances, don't be shy about posting a question in the appropriate Discussion topic or PMing someone if you have trouble with the handwriting or are not sure how to handle something.
Don't forget that we can now add our own comments via the Event Input Wizard with Transcriber Comment. There is a Weather Record subtype if you want to add a note about data in the weather grid. Be sure to click in one of the blue boxes to enter the comment or it will not be saved.

Finally, Kevin says that there are tests that can be applied after transcription to find and remove the larger errors. He suspects that log keepers are a bigger source of errors than transcribers.


As in the past, please transcribe all weather records.

Gil Compo says every observation makes a difference everywhere and that this is especially true for the West Coast during the time period of these logs. There is very limited data, so our transcriptions will improve the reanalysis of the weather (storms, high winds, and precipitation) and how it changes over time. They may also help green energy. There is considerable interest in the probability distribution of winds for wind energy siting.

Even the months of data when they are in port are valuable. They can help to calibrate data from stations within up to 1000 km, and they provide greater certainty than a single station can give. Also, not all ports have stations.
Title: Re: Project Updates
Post by: Randi on 06 November 2019, 13:41:51
New spreadsheet available

We have just released a spreadsheet with many improvements to make transcribing, and processing, easier!
You can access it from the Discussion topic for your ship-year.
The Location (http://forum.oldweather.org/index.php?topic=4690.msg144395#msg144395) section of the Transcribing guide has been updated and Sailing Information is being updated.
Title: Re: Project Updates
Post by: Randi on 08 December 2019, 15:36:16
Kevin hopes to scan Bear 1920 next week!!!


For those in a mood for warmer weather - and typed logs - there is a special project:
Quote from: Kevin
As mentioned earlier, Gil has asked if we could do an express transcription of some specific storm events for testing 20CR Version 3, just out.

The first case would be 3 ships on the China station in 1930-31. The period would be leading up to the great Yangtze flood and culminating in August 1931.

https://en.wikipedia.org/wiki/1931_China_floods


The dates to transcribe would be Sept 1, 1930 to August 31, 1931. In the attached spreadsheet of new ships loaded onto the National Catalog, these are:

USS Helena
https://catalog.archives.gov/id/148800961
https://catalog.archives.gov/id/148801445

USS Monocacy II
https://catalog.archives.gov/id/148838801

USS Panay II
https://catalog.archives.gov/id/148865382
https://catalog.archives.gov/id/148866445

https://en.wikipedia.org/wiki/USS_Helena_(PG-9%29
https://en.wikipedia.org/wiki/USS_Monocacy_(PG-20%29
https://en.wikipedia.org/wiki/USS_Panay_(PR-5%29
Title: Re: Project Updates
Post by: Hanibal94 on 08 December 2019, 17:06:14
Thanks for the news, Randi - both the Bear and the special project.

I think I will take a break from the Northland to go after the latter - would be nice to have something different for a change!
Title: Re: Project Updates
Post by: ggordon on 08 December 2019, 20:09:16
I'd be interested in taking a break and working on the data for China too.  It sounds like that may be more urgent at the moment than another year of Northland data.  I'm willing to work on whichever is deemed more urgent.
Title: Re: Project Updates
Post by: Pommy Stuart on 08 December 2019, 20:44:13
I could do any of the projects needed in place of Northland 1933.
Title: Re: Project Updates
Post by: Randi on 08 December 2019, 20:50:45
They are open to all.

However, given that they are typed logs Kevin mentioned that they would be good for the new transcribers we hope to get from upcoming publicity.

How about if you use Helena as a vacation and beta test of the new system and reserve the other two ships for new transcribers?
Title: Re: Project Updates
Post by: ggordon on 08 December 2019, 21:05:46
From my experience as a fairly recent newcomer it was a major help to start with typed logs.  So consider me available if new transcribers don't appear.  Otherwise I'll continue the struggles with my current log keepers.   ::)
Title: Re: Project Updates
Post by: Pommy Stuart on 09 December 2019, 00:45:32
Gordon
If Randi does not get starters which Helena do you want to do, 1930 or 31 and I will do the other.
If anybody can break it it's me,  :o and If anybody can fix it it's you.  ;D

Went to look at the Spreadsheet, it is still locked.
Title: Re: Project Updates
Post by: ggordon on 09 December 2019, 01:52:52
I'll wait to see what Randi wants to do.  If there are enough volunteers, she may want to assign individual months rather than years so as to get the project completed faster.
Title: Re: Project Updates
Post by: studentforever on 09 December 2019, 04:45:08
Had a quick look some pages on the link and there are some RN ships there as well. Those logs would be in the National Archives if they could help the data analysis
Title: Re: Project Updates
Post by: Randi on 09 December 2019, 09:49:29
Thanks!
I passed your comment on to Kevin just in case he doesn't see it here.
Title: Re: Project Updates
Post by: Randi on 11 December 2019, 13:08:40
Helena is now available to all!

It will help test the new spreadsheet and give Michael, Kevin and Gil a chance to test the system ::)
Maybe some new locations for Matteo too!

Note that pressure values are particularly important.
Title: Re: Project Updates
Post by: Pommy Stuart on 12 December 2019, 00:23:04
Helena. 1930
I am halfway through Sept with some files uploaded.
Not much chance of seeing Ice where we are.  ::)

Very easy log to do. Even I can read the writing.   :o
How far do you want me to go before you hand it over to Newbies?

Title: Re: Project Updates
Post by: Randi on 12 December 2019, 00:43:50
For now, why don't you and anyone else who wants a break from Northland finish Helena (1930)?

We have a lot of new members, but I don't know how many will want to work on the federal ships.
Title: Re: Project Updates
Post by: Pommy Stuart on 12 December 2019, 01:56:50
OK
Title: Re: Project Updates
Post by: Hanibal94 on 12 December 2019, 11:57:10
Alright, I reserved November and will start work on it once I finish my current Northland month.

How urgent is this, Randi? You mentioned "express transcription" in your original post.
If the scientists REALLY want the data, it might be better to let us veterans chop our way through these logs - especially since some of us might have more time with the holidays coming up.
Just a suggestion.
Title: Re: Project Updates
Post by: Randi on 12 December 2019, 12:09:42
Well, Kevin said both express and good for new transcribers because they are typed in his email to me ;D
I would hate to start my transcribing with Northland's logs!
Let's see what the status is once Helena 1930 is done.
We have a lot of new transcribers...