D4Science Infrastructure

Gianpaolo Coro wrote:

I ask to our system admins: do you see memory leaks?

I should investigate while the compilation is running. It's possible that it consumes all the available memory but I have to check. From the logs nothing strange happened.

Dear Andrea,
is there any progress on this issue please?
Thank you...

TAHA IMZILEN wrote:

Dear Andrea,
is there any progress on this issue please?
Thank you...

@taha.imzilen@ird.fr I need to know when you try to compile that document. Because no error is logged when a compilation fails, I have to watch what's happening while the compilation is running. If you can tell me when you can start another one, I'll check what happens.

@andrea.dellamico@isti.cnr.it , I'm off work this week, we can try that next week. I will keep you informed for this.
Thank you

@taha.imzilen@ird.fr perhaps you should just share the sharelatex project with Andrea...

Dear @andrea.dellamico@isti.cnr.it , can we try to compile the document this afternoon?
as @julien.barde@ird.fr suggests, I shared the project with you, this can make things easier.

I can see the project. I'll try to compile it later in the afternoon.

The compiler times out while waiting something that I've not identified yet. You are trying to process more that a GB of data, btw:

 ls -ltrah
total 1.5G
drwxr-xr-x 4 www-data www-data  118 Oct 17 13:39 ..
drwxr-xr-x 2 www-data www-data   27 Oct 17 13:39 oldshit
drwxr-xr-x 2 www-data www-data 4.0K Oct 17 13:39 figures
drwxr-xr-x 2 www-data www-data 4.0K Oct 17 13:55 cache
-rw-r--r-- 1 www-data www-data    0 Oct 17 13:56 output.out
-rw-r--r-- 1 www-data www-data   63 Oct 17 13:56 output.xwm
-rw-r--r-- 1 www-data www-data 1.1K Oct 17 13:56 output.aux
-rw-r--r-- 1 www-data www-data  30K Oct 17 13:56 output.fls
-rw-r--r-- 1 www-data www-data 3.9K Oct 17 13:56 output.synctex.gz
-rw-r--r-- 1 www-data www-data  29K Oct 17 13:56 output.fdb_latexmk
drwxr-xr-x 3 www-data www-data   17 Oct 17 13:56 .cache
-rw-r--r-- 1 www-data www-data  29K Oct 17 14:29 elsarticle-num.bst
-rw-r--r-- 1 www-data www-data  27K Oct 17 14:29 elsarticle-num-names.bst
-rw-r--r-- 1 www-data www-data  11K Oct 17 14:29 test_compile.Rtex
-rw-r--r-- 1 www-data www-data 5.5K Oct 17 14:29 numcompress.sty
-rw-r--r-- 1 www-data www-data  19K Oct 17 14:29 fad_drift_article_pnas_appendix.Rtex
-rw-r--r-- 1 www-data www-data 100K Oct 17 14:29 fad_drift_article_pnas.Rtex
-rw-r--r-- 1 www-data www-data 7.4K Oct 17 14:29 workspace_interaction.r
-rw-r--r-- 1 www-data www-data  18K Oct 17 14:29 functions_FADsDrifters_oscar.R
-rw-r--r-- 1 www-data www-data 3.2K Oct 17 14:29 dist_spat_data_with_clean_labels.R
-rw-r--r-- 1 www-data www-data  770 Oct 17 14:29 steptime_freq.R
-rw-r--r-- 1 www-data www-data 3.9K Oct 17 14:29 smooth_scatterplot_UVspeed_greens.R
-rw-r--r-- 1 www-data www-data 7.7K Oct 17 14:29 speed_map.R
-rw-r--r-- 1 www-data www-data 4.0K Oct 17 14:29 ErrorMov_freq.R
-rw-r--r-- 1 www-data www-data 2.7K Oct 17 14:29 ErrorMov_freq_manu.R
-rw-r--r-- 1 www-data www-data 3.0K Oct 17 14:29 subData_by_Longhurst.R
-rw-r--r-- 1 www-data www-data 5.1K Oct 17 14:29 spatial_distribution_of_correlation_func.R
-rw-r--r-- 1 www-data www-data 159K Oct 17 14:29 fad_drift.bib
-rw-r--r-- 1 www-data www-data  18K Oct 17 14:29 pnas-new.bst
-rw-r--r-- 1 www-data www-data  28K Oct 17 14:29 elsarticle-harv.bst
-rw-r--r-- 1 www-data www-data  26K Oct 17 14:29 elsarticle.cls
-rw-r--r-- 1 www-data www-data  15K Oct 17 14:29 pnas-new.cls
-rw-r--r-- 1 www-data www-data 1.5K Oct 17 14:30 pnasresearcharticle.sty
-rw-r--r-- 1 www-data www-data 664K Oct 17 14:30 fig_drifter_fad_design.png
-rw-r--r-- 1 www-data www-data 2.4M Oct 17 14:30 Longhurst_world_v4_2010.zip
-rw-r--r-- 1 www-data www-data  164 Oct 17 14:30 Longhurst_world_v4_2010.sbx
-rw-r--r-- 1 www-data www-data  652 Oct 17 14:30 Longhurst_world_v4_2010.sbn
-rw-r--r-- 1 www-data www-data  145 Oct 17 14:30 Longhurst_world_v4_2010.prj
-rw-r--r-- 1 www-data www-data 5.7K Oct 17 14:30 Longhurst_world_v4_2010.dbf
-rw-r--r-- 1 www-data www-data 2.9M Oct 17 14:30 Longhurst_world_v4_2010.shp
-rw-r--r-- 1 www-data www-data  532 Oct 17 14:30 Longhurst_world_v4_2010.shx
-rw-r--r-- 1 www-data www-data 8.3K Oct 17 14:30 Longhurst_world_v4_2010.shp.xml
-rw-r--r-- 1 www-data www-data 2.4M Oct 17 14:30 cwp_grid_shape.zip
-rw-r--r-- 1 www-data www-data  256 Oct 17 14:30 cwp_grid.prj
-rw-r--r-- 1 www-data www-data  11M Oct 17 14:30 cwp_grid.dbf
-rw-r--r-- 1 www-data www-data 9.3M Oct 17 14:30 cwp_grid.shp
-rw-r--r-- 1 www-data www-data 560K Oct 17 14:30 cwp_grid.shx
-rw-r--r-- 1 www-data www-data 163M Oct 17 14:30 dataFADs_atlantic_2008_2014_last.zip
-rw-r--r-- 1 www-data www-data 403M Oct 17 14:30 dataFADs_atlantic_2008_2014_last.csv
-rw-r--r-- 1 www-data www-data 241M Oct 17 14:33 dataFADs_indian_2008_2014_last.zip
drwxr-xr-x 6 www-data www-data 4.0K Oct 17 14:33 .
-rw-r--r-- 1 www-data www-data 658M Oct 17 14:33 dataFADs_indian_2008_2014_last.csv

It seems that I had success to compile the test document. I tried as a draft first, and then in regular mode.
I then tried to compile fad_drift_article_pnas.Rtex, also in draft mode. It failed, the error starts here:

LaTeX Info: Redefining \microtypecontext on input line 9.

Package microtype Warning: `draft' option active.
(microtype)                Disabling all micro-typographic extensions.
(microtype)                This might lead to different line and page breaks.


! LaTeX Error: Environment knitrout undefined.

See the LaTeX manual or LaTeX Companion for explanation.
Type  H <return>  for immediate help.
 ...

l.15 \begin{knitrout}

Your command was ignored.
Type  I <command> <return>  to replace it with another command,
or  <return>  to continue without it.


! Package xcolor Error: Undefined color `fgcolor'.

See the xcolor package documentation for explanation.
Type  H <return>  for immediate help.
 ...

l.16 ...}{rgb}{0.969, 0.969, 0.969}\color{fgcolor}
                                                  \begin{kframe}
Try typing  <return>  to proceed.
If that doesn't work, type  X <return>  to quit.


! LaTeX Error: Environment kframe undefined.

See the LaTeX manual or LaTeX Companion for explanation.
Type  H <return>  for immediate help.
 ...

I'll try to upload the full log.

I've also found a sharelatex document that gives hints on how to debug compilation timeouts: https://www.sharelatex.com/learn/Debugging_Compilation_timeout_errors

I was trying to see if the compilation completes after the web interface fails, but a compilation of the test file was triggered and it killed the other one. I'll retry tomorrow.

@taha.imzilen@ird.fr can you find anything useful to you into the logs? If want to repeat the procedure, compiling the document as draft produces a readable log file almost at every run.

@andrea.dellamico@isti.cnr.it , I am still looking into the logs, I have no idea what's the matter. For the R computation error, i checked in Rstudio and i didn't get any error.

I confess I am a little lost, can we first compile only the first part of the document (up to line 378) to read all the data without compiling the R computations ? it may help to understand what's happen!

I have shared the new 'fad_drift_compilation' project with you and I'm going step-by-step. I'm stuck at line 90 because there is something wrong with the zip file in your /Home/taha.imzilen/Workspace/FAD_Drifter/ folder and the readOGR function fails. Could you give it a look please?

I would go in the same way adding one piece of code at time and seeing if there are warnings or errors. When one of these is met, we can check what's happening.

Hi Gianpaolo,
thanks for your help!
I checked, the function downloadFileWS download an empty zip file.
file.exists('Longhurst_world_v4_2010.zip') == TRUE
file.size('Longhurst_world_v4_2010.zip') == 0

Could you upload it again on the Workspace, please?

Hi Gianpaolo,
ok, it's done.
Thank you!

Hi Taha, the issue was that the R code was not using the workspace_interaction.r source, but only the ShareLatex default one which is obsolete (@andrea.dellamico@isti.cnr.it is it possible to update it?). If you notice, in the CheckWSInteraction example previously shared with you, I had explicitly sourced this file.

Thus, in order to solve the zero-size downloading issue, it was sufficient to add these lines to the code (I have modified the original fad_drift project too at lines 56-59):

dataDriftersWS <- '/Home/taha.imzilen/Workspace/FAD_Drifter'
username<<- 'taha.imzilen'
token <<- '<your token obfuscated>'
source("workspace_interaction.r")

However, the compilation process of the overall ShareLatex document time outs due to the huge amount of prerequisite software that is downloaded before the compilation. I think the online ShareLatex technology was not designed for this kind of application (you are too ahead of the times :-)).

I see two alternative ways to go forward:

1 - split the document into several documents or projects;
2 - collect all the auxiliary scripts and material in one zip file and just do one download before the compilation.

yes i agree! I already thought about doing this and creating a single zip to download.
I will change that and I will keep you informed.
thank you

Gianpaolo Coro wrote:

Hi Taha, the issue was that the R code was not using the workspace_interaction.r source, but only the ShareLatex default one which is obsolete ( is it possible to update it?). If you notice, in the CheckWSInteraction example previously shared with you, I had explicitly sourced this file.

Done, I also ensured that it will be always up to date.

Hi @gianpaolo.coro@isti.cnr.it ,
I tried to collect all scripts and material in one zip file and do one download before the compilation and it works much better on Rstudio but still not work on sharelatex. i get the same message error :
'Server Error Sorry, something went wrong and your project could not be compiled. Please try again in a few moments.'
I will split the main document into several documents by using the subfiles package and let you know as soon as possible .
PS: sorry for the delay, i didn't see your message and i didn't receive any notification.
Thank you