Tao Effect Forums

Espionage => Espionage 3 => Topic started by: nomolo on October 04, 2014, 12:48:30 AM

Title: Unlocking large folders in 3.6.1
Post by: nomolo on October 04, 2014, 12:48:30 AM
When I unlock my large Mail folder (65 GB) with version 3.6.1. the slider in the Espionage window is very slow to reflect the unlocked status. It remains grayed out, even though Mail has been unlocked, launched and is functioning normally (see screenshot). While its grayed out, Mail can't be locked. Clicking the "i" updates the button to its correct state. Otherwise, it remains grayed out for several minutes.   

On the other hand, I can now sometimes lock this large Mail folder without first manually quitting Mail, which hadn't been possible with 3.6.0.
Title: Re: Unlocking large folders in 3.6.1
Post by: zsolt on October 04, 2014, 02:30:13 PM
Hello. please allow me to test this and I will come back with my findings.

Rgds
Zsolt
Title: Re: Unlocking large folders in 3.6.1
Post by: zsolt on October 05, 2014, 03:48:54 AM
I tried to reproduce it, but failed, both the lock/unlock of a large folder, and locking folder while there is a document open.
From my experience with Espionage, the speed at which the folder unlocks/locks does not depend on the size of the folder, at least not in a visible way.
I do recall that sometimes when you try to mount a big disk image, like OS X install disk image or such, there is a window saying "verifying disk image", so maybe this is the time during which the folder is in this undefined state. I will check with devs if this is possible.

For the folder allowing to lock while a document is open, if I try that, I get a notification with OS error -47 which is the equivalent of "file busy" error.
Now, this is of course if there is a file open. You did not say that there was an open file, but that the Mail was running. I can hardly imagine that a running Mail does not keep any single file open, but then again, all is possible in some rare circumstances.

So what I would kindly ask you:
- re big disk image, I will check with devs about details, and you please try to reproduce it again, if you manage, then please open the console application, select system log and later "all messages" and see if there are some recent espionage related lines.
- re possibility to lock the folder while a file is open, can you please test with some other folder, other then Mail. If you do manage to reproduce it, then let me know. If you can reproduce it with mail, then while the folder is locked and Mail running, run the following command from the terminal

lsof | grep -i mail >lsof.out, and send me the lsof.out file.

Thanks
Zsolt
Title: Re: Unlocking large folders in 3.6.1
Post by: nomolo on October 05, 2014, 07:22:53 AM

Thanks Szolt

I have a folder action to quit Mail before locking it. So I assumed that what I observed (clicking the "Lock" button causes Mail to quit and then lock) is the intended behaviour. Isn't that right? 

(I have a similar folder action with BusyCal. Just a single folder action: Quit BusyCal.  This allows me to quit and then lock BusyCal while its running, which it does successfully.)

More often, when I try and lock Mail while it is running, Mail just quits but fails to lock, presumably because other files associated with Mail are still open. I've just tried it again and this is the output:

05/10/2014 15:14:18.126 Espionage[393]: ERROR (+[DiskMaster unmountDiskForFolderFace:mountpoint:forcefully:updateUI:error:]:533): failed because: unlockStatus != noErr
05/10/2014 15:14:18.126 Espionage[393]: ERROR (-[LockFolderTask go:]:84): failed to unmount '/Users/mf/Library/Mail': Error Domain=NSOSStatusErrorDomain Code=-47 "The operation couldn't be completed. (OSStatus error -47.)" (fBsyErr: File is busy (delete))
05/10/2014 15:14:18.126 Espionage[393]: INFO: Posted critical message: Failed to lock 'Mail': The operation couldn't be completed. (OSStatus error -47.)


I will send you the requested console outputs when I reproduce the phenomenon!
Title: Re: Unlocking large folders in 3.6.1
Post by: nomolo on October 05, 2014, 09:50:54 AM
I have now locked Mail again while it was running with the Inbox open. I have also observed the greyed out button after unlocking Mail. I repeated each 3 times but nothing related to Espionage has come up in the console.
Title: Re: Unlocking large folders in 3.6.1
Post by: zsolt on October 05, 2014, 10:04:00 AM
Hello, the mail is a bit tough to play with as it is not a simple application i.e. quite often it will not quit right away when you try to quit it.
And yes, the folder action as you set it, has the purpose to quit it before locking it.

So let me just summarise it quickly:

- the fail to lock is explainable, as Mail will often fail to quit right away, even if you try to quit it by hand. This is because it tries to close the connections first and clear up the caches before disconnecting from Mail servers. And sometimes, even after it quit, some files remain open, this you can check with the lsof command I sent you before. There is nothing much we can do here, except implement some kind of retry and inform user about this, I might file this as feature request if you are interested.

- the grayed out button happens only when you unlock mail (which is again handled through folder action - unlock the folder, launch mail), correct?
If so, I would like to see if you get the same behaviour if we get rid of the folder action.
Pls remove it, then unlock the folder first, and see if you will (ever) get this grayed out button.
You do not even need to launch mail after that. If we test in this way, it is simply a matter of unlocking a big folder.

Please let me know how it went.

Thanks
Zsolt
Title: Re: Unlocking large folders in 3.6.1
Post by: nomolo on October 05, 2014, 10:37:05 AM
With the folder actions in place, the greyed out button changes to white 55 seconds after clicking 'Unlock'

Without the folder actions (ie deleting 'Open Mail' and 'Unlock Folder com.apple.mail') the greyed out button changes to white 4 minutes after clicking 'Unlock'
Title: Re: Unlocking large folders in 3.6.1
Post by: nomolo on October 05, 2014, 10:41:21 AM
Zsolt

Sorry the above figures are the wrong way round!  Its 4 minutes to unlock with the folder actions; 55 seconds without them!
Title: Re: Unlocking large folders in 3.6.1
Post by: greg on October 05, 2014, 03:34:48 PM
I've created an issue for this in our internal issue system and assigned it a high priority (for next release).
Title: Re: Unlocking large folders in 3.6.1
Post by: nomolo on October 05, 2014, 03:59:20 PM
Thanks!
Title: Re: Unlocking large folders in 3.6.1
Post by: zsolt on October 06, 2014, 11:40:19 AM
Hello, as Greg mentioned, we are testing this internally, trying to reproduce it.
But in the mean time, IF you wish to spend some more time helping us out, then here is what you can do:

- you do mention version 3.6 at the beginning of your post. The problem with grayed out button, did it happen on 3.6 too? Do you have the same folder setup now and when you had version 3.6 running?
- is this all on Yosemite or on Maverick
- can you start up task manager, from the view menu, select All processes, and in the search window of the activity monitor enter "hdi" without quotes.
See if you have any hdiutil process running, probably not. Now keep the window visible, and unlock the problematic folder, there should one hdiutil process appear. In about 5 seconds after the folder is unlocked, it should go away. Let me know at which point did it go away, 5 seconds after you unlocked the folder, or 5 seconds after the button got out from the grayed state, or at some other moment.

Thanks
Zsolt
Title: Re: Unlocking large folders in 3.6.1
Post by: nomolo on October 07, 2014, 02:17:52 AM
I'm using Mavericks, 10.9.5

I'm not aware of having seen this problem with 3.6.0. If there was a delay, it would have been very short and I would have assumed it was normal for a large file. I saw nothing like the delay of several minutes I'm getting with 3.6.1

I have added some new encrypted folders since 3.6.0 but the Mail folder which is causing the problem has not changed, it was encrypted under 3.6.0 as well as 3.6.1

Running the procedure you suggested: hdiutil pops us then disappears about 5 seconds after Mail unlocks. It doesn't hang about until the button goes from gray to white.

Let me know if I can run any other kind of test that might help.
Title: Re: Unlocking large folders in 3.6.1
Post by: zsolt on October 07, 2014, 05:21:38 AM
Thanks a lot, now we know in which direction to test.
I will let you know if we find something.

Rgds
Zsolt
Title: Re: Unlocking large folders in 3.6.1
Post by: greg on October 07, 2014, 08:57:58 PM
Dear nomolo,

I think we're going to need a few days, and then we'll have a build ready for you to test and verify that the problem has been fixed. Sorry about this, just right now there are too many things going on because of Yosemite (for example, Ernesto, the other dev, needs to update his apps for Yosemite) and other urgent items.

In the meantime, is there some type of workaround that you can use? Perhaps temporarily removing the app actions might help?

Kind regards,
Greg
Title: Re: Unlocking large folders in 3.6.1
Post by: nomolo on October 08, 2014, 12:07:21 AM
Thanks Greg - no problem.
Title: Re: Unlocking large folders in 3.6.1
Post by: erndev on October 20, 2014, 03:13:51 PM
Dear nomolo

I'm Ernesto, one of the developers of Espionage.
I'd like to ask if it's possible to have a log of the application while you unlock the folder that is taking so long. It might help us fix the issue.

In order to get the log, open the Console app, and choose "All Messages". There you'll be able to see the relevant logs of Espionage, and copy them in order to send it to us.


cheers
Ernesto
Title: Re: Unlocking large folders in 3.6.1
Post by: greg on October 20, 2014, 04:14:23 PM
FYI, Ernesto has been trying to reproduce this but so far he hasn't been able to. One likely reason I'm guessing is b/c he's using 60GB worth of fake data (not real Mail). So we're going to try to modify Espionage to insert a fake delay upon launching Mail and see if we can reproduce that way.

In the meantime, please let Ernesto know the answer to his question, and also what operating system you're using (e.g. 10.9, 10.10, etc.). Sorry if you've already given this information to us, but I didn't see it in this thread!
Title: Re: Unlocking large folders in 3.6.1
Post by: nomolo on October 20, 2014, 05:00:08 PM
I'm sorry this is causing you so much work and really impressed with the lengths you're going to.

I'm using Mavericks 10.9.5 and Espionage 3.6.1

I've just unlocked Mail twice. The first time I waited for the slider button to go from grey to white on its own. There were no Espionage entries in the the console log at all. The second time, rather than waiting the full 4 or 5 minutes for the button to change color, I clicked on the "i" button next to the slider, which prompts the button to go white, and this entry appeared:

21/10/2014 00:45:20.475 Espionage[182]: ERROR (-[StatFolderTask go:]:56): failed to calculate folder size for '/Users/mf/Library/Mail': Error Domain=Espionage Code=1000 "User canceled size calculation" UserInfo=0x60000447ae40 {NSLocalizedDescription=User canceled size calculation}

There were a lot of Mail entries in the console during this period. Are they of any interest?


Title: Re: Unlocking large folders in 3.6.1
Post by: greg on October 22, 2014, 03:39:29 PM
OK, we haven't been able to reproduce this even with the fake delay for Mail.

So, Ernesto created a custom build for you and should have sent you a private message with a link to it. This build will output much much more logging messages to the system log, which will help us figure out what's going on over on your computer.

Just a heads up:

1. There's a possibility that we'll release 3.6.2 prior to fixing this problem in order to fix more urgent bugs.
2. Please do not post the logs from the Console application here. In order to protect your privacy, please email them directly to us instead (and make sure to reference this thread in your email): support@taoeffect.com