Author Topic: Team render issues  (Read 2160 times)

2019-02-08, 16:58:50
Reply #30

kizo

  • Active Users
  • **
  • Posts: 28
    • View Profile
yes Tom the no benefit in using the server is base on an actual test...one of the numerous tests we did. the render time is again almost same as on a single 2990wx and the machines rendering have way more power combined

the fewer machines tests were done with 30/100  as I wrote above.

Im glad to hear the devs are thinking about the issues as well as Im very glad on your help. Im only worried how will I manage to work on daily bassis once the betas stop working.
At this point I can see  any improvement will need time. I also see I will not be able to use the nodes that I bought the licences for and I will not be able to use all the costly hardware I have.

very unfortunate and seems I will be forced to look for alternative rendering solution which is a shame as I love Corona very much.

The only other thing I can do is make the 10G network.That is an expensive operation so before investing I would like to know would that resolve the issues for sure?

kizo
« Last Edit: 2019-02-08, 17:07:10 by kizo »

2019-02-08, 17:32:44
Reply #31

jojorender

  • Active Users
  • **
  • Posts: 60
    • View Profile
I recently reported client disconnects with just 3 clients. At the time when TR server had the memory leak issue I only used 1 client and that also had frequent disconnects.
This happened in automatic and in manual mode. Can’t imagine that 1 client can cause network congestion, but who knows? 

Is increasing the interval and/or # of retries when frame sync fails a way to give that “keep alive car” a chance to sneak onto the highway?

As kizo mentioned, setting up a 10Gb network is a considerable expense, even for small networks.
Cheapest 8-port switch cost about $600, + NIC’s, cables, etc.
About $1k+ to network 4 machines, not knowing if that will really solve the problem.
I already spend a lot of money on RAM, trying to fight the TR memory leak. LOL.
I learned my lesson.

Since you know your customers best, maybe you can survey all your 10+ nodes customers (especially those that have nothing to report) about their TR experience and network implementation and pass along your findings?

Also, as Stefan (former Mr. VrayforC4D I believe) points out here https://corona-renderer.com/forum/index.php?topic=23033.msg140809#msg140809 TR is flawed and he advocates for a real DR solution like implemented for max.
Maybe you can pick his brain, since he must have a pretty good insight into the inner workings/ shortcomings of TR.

We love you Corona!
MacOs 10.13.6
C4D Studio R18.057

2019-02-08, 17:35:31
Reply #32

TomG

  • Corona Team
  • Active Users
  • ****
  • Posts: 2737
    • View Profile
You mention "how will I manage to work on daily bassis once the betas stop working".. that made me wonder, how were you working before the betas stop working? Does this mean Team Render with lots of clients etc. was working fine with the betas, and now is no longer working with the commercial licensed release? That would be unusual if so and I can't think of a reason for it, but would be useful information to know.

On the 10 gig network question, unfortunately we can't offer any guarantees on that, sorry.  While it may assist, as it can handle much more traffic, since we haven't been able to test on a 1 Gig versus a 10 Gig with 10 Clients and compare the difference ourselves, we can't offer any sort of guarantee on it resolving the issue, or on at what point the issue would appear again (at certain packet sizes or image resolutions).

2019-02-08, 17:51:02
Reply #33

kizo

  • Active Users
  • **
  • Posts: 28
    • View Profile
Im working without TR.In fact I never use it as it always had issue in my tests.Also I have never used it as other engines we use(d) dont require it.
I have Cinema installed on each machine and corona beta.so I just render on a single machine....big stills (6000x4000) overnight or range of animations on each machine at the same time. works excellent.
Now that its a product with license I wanted to avoid buying 10 licences as its a heavy price.


great points jojorender !
I agree a dedicated DR solution would be best choice. Also it would be really great to hear from anyone using TR with success in general even with fewer machines.
« Last Edit: 2019-02-08, 23:31:28 by kizo »

2019-02-11, 10:16:37
Reply #34

HFPatzi

  • Active Users
  • **
  • Posts: 92
    • View Profile
Hey there Teamrenderers ;)

I guess i would be one of a few corona users with more then 2 machines. Is it helpful for you, if i keep you updated about my refresh/size settings and what's happening during TR? Although i don't have time to do much tests right now since I'm cluttered with renderjobs at the moment. But as I said, I can keep you updated what's happening with certain settings during rendering. And I probably will replicate some of the settings kizo used for his tests. Thank you very much, btw for doing all the testing!!!

My renders will mostly be 9000 x 6000 Pixels @72dpi. But in the near future I'll have some really huge ones. About 15000 x 9000 pixels. Would be pretty interesting I guess :)

Here are my Specs:

Workstation: Intel i9-7940X @3,1GHz, 64GB DDR4 RAM, M2 SSD (1TB), Win 10

Renderclients:

8 x Intel i7-7700 @3,6GHz, 32GB DDR4 RAM, Win 10
1 x Mac Pro (Late 2013), Intel Xeon E5 @3,5GHz, 32GB DDR3 RAM, OSX 10.12.6
1 x MacPro (Mid 2012), 2 x Intel Xeon @2,6GHz, 16GB DDR3 RAM, OSX 10.12.6

(Company)Network:

1 x HP2810-24G Switch
2 x Cisco SG 11D-08 Switch

My workstation is connected directly to the HP Switch as well as both Mac-Clients. Due to lack of free Ports on the HP-Switch, the 8 Windows Clients are splitted (4 each) to the 2 cisco switches wich are connected to the HP-Switch.
Not the greatest setup I know, but the only one that works right now without buying new, expensive hardware.
Also upgrading to a 10Gbit network will be pretty expensive for a small company like ours, where CGI isn't the main income.
In the past I experienced dropouts from basically every client in pretty much random order. Sometimes I had a feeling that only every other Windows client stopped rendering. But that's not for sure.

I'm truly sorry, that i can't provide any testing right now. Will do for sure, if i have more time in the future, since I love corona and want to help improve this great product.
If you have any question regarding my setup, feel free to ask!

Greetings,
Moritz


« Last Edit: 2019-02-11, 22:16:21 by HFPatzi »

2019-02-11, 19:54:53
Reply #35

TomG

  • Corona Team
  • Active Users
  • ****
  • Posts: 2737
    • View Profile
Hi!

Thanks for the offering of testing when time allows Moritz, we welcome any data you are able to send.

@Kizo, thank you for all your testing too. When time allows, we'd be interested to know if you open some large scene from the C4D Content Browser and rendering it using a native C4D render engine, using Team Render. This would be useful to know, to see if the problem lies with Team Render in general, or if our specific implementation of Team Render has added some problems that don't occur with inbuilt engines and TR. Sorry to ask for more testing, but the information is invaluable to us!

And of course our own tests and investigations continue internally, too.

Thanks all!

2019-02-11, 22:08:09
Reply #36

HFPatzi

  • Active Users
  • **
  • Posts: 92
    • View Profile
Guess what? There was a "small" timeframe this evening, so I took the oportunity ;)

I always rendered the same scene and same size: 14763 x 8858 pixels with different "amounts" of clients.
Every combination was rendered with two different network settings: 50s / 50MB and 100s / 100MB

Here are the results:


WS Only:
Network settings: -
Rendertime: 1h 00min
Client dropouts: 0
Clients finished: 1
Console error: -
Dropout time: -


WS + 2 Mac clients:
Network settings: 50s/50MB
Rendertime: 0h 26min
Client dropouts: 0
Clients finished: 3
Console error: -
Dropout time: -


WS + 2 Mac clients:
Network settings: 100s/100MB
Rendertime: 0h 25min
Clients dropouts: 0
Client finished: 3
Console error: -
Dropout time: -


WS + 2 Mac / 2 Win clients:
Network settings: 50s/50MB
Rendertime: 0h 17min
Client dropouts: 0
Clients finished: 5
Console error: -
Dropout time: -


WS + 2 Mac / 2 Win clients:
Network settings: 100s/100MB
Rendertime: 0h 17min
Client dropouts: 0
Clients finished: 5
Console error: -
Dropout time: -


WS + 2 Mac / 4 Win clients:
Network settings: 50s/50MB
Rendertime: 0h 16min
Client dropouts: 1 Win / 0 Mac
Clients finished: 6
Console error: Frame synchronization failed: Communication Error
Dropout time: While Finishing and Sending back chunks. Rendering was finished.


WS + 2 Mac / 4 Win clients:
Network settings: 100s/100MB
Rendertime: 0h 14min
Client dropouts: 0 Win / 1 Mac
Clients finished: 6
Console error: Frame synchronization failed: Communication Error
Dropout time: While Finishing and Sending back chunks. Rendering was finished.


WS + 2 Mac / 6 Win clients:
Network settings: 50s/50MB
Rendertime: 0h 14min
Client dropouts: 3 Win / 0 Mac
Clients finished: 6
Console error: Frame synchronization failed: Communication Error
Dropout time: While Finishing and Sending back chunks. Rendering was finished.


WS + 2 Mac / 6 Win clients:
Network settings: 100s/100MB
Rendertime: 0h 14min
Client dropouts: 3 Win / 1 Mac
Clients finished: 5
Console error: Frame synchronization failed: Communication Error
Dropout time: While Finishing and Sending back chunks. Rendering was finished.


WS + 2 Mac / 8 Win clients:
Network settings: 50s/50MB
Rendertime: 0h 14min
Client dropouts: 3 Win / 2 Mac
Clients finished: 6
Console error: Frame synchronization failed: Communication Error
Dropout time: While Finishing and Sending back chunks. Rendering was finished.


WS + 2 Mac / 8 Win clients:
Network settings: 100s/100MB
Rendertime: 0h 14min
Client dropouts: 6 Win / 0 Mac
Clients finished: 5
Console error: Frame synchronization failed: Communication Error
Dropout time: While Finishing and Sending back chunks. Rendering was finished.



I will also upload an excel sheet with the test results, for better readability and the console output of my workstation via your private uploader. I will edit this post whith the uploaded filename, when done.
What also conserns me is that there is no decrease in rendertime aftrer a certain number of clients. Not sure if this is caused by the increasing number of dropouts.
The main Problem i have with TR right now is, that one has to do a lot of tests to find out the perfect render settings/ number of clients for every project. Almost all time advantage you have with TR is eaten up by finding the best settings for each project.
And, on a side note: Before corona, I rendered with Physical Renderer and also via TR (same image size and same kind of scenes). And as far as I can remember, i never had any dropouts back then.

So much for my testing today ;)
If there are any questions, feel free to ask!

Greetings,
Moritz

P.s.: the Workstation's console output was uploaded via your private uploader uder the name: 1549919693_Workstation-console-output.txt

========

EDIT: Forgot to mention that i set a pass limit of 15 for my tests.
« Last Edit: 2019-02-12, 09:53:48 by HFPatzi »

2019-02-12, 08:25:52
Reply #37

HFPatzi

  • Active Users
  • **
  • Posts: 92
    • View Profile
Just out of curiosity, I did one last test this morning with all clients and settings as follows: 150s/50MB.

Rendertime was 13min 30s and again I had 5 dropouts while finishing and sending back chunks.
Here are 2 example-console outputs of the renderclients. They are slightly different, so I don't know if thats interesting for you:

Code: [Select]
Rendering frame 0 of job 'pv'
2019/02/12 07:54:21  [Corona4D] Parsing scene
2019/02/12 07:54:21  [Corona4D] ====RENDER STARTED====
2019/02/12 07:54:21  [Corona4D] Core build timestamp: Jan 17 2019 14:36:24
Asset 'XXXXX.psd' erstellt
Lade Asset(s) herunter
Asset 'XXXXX.tif' erstellt
Asset 'XXXXX.tif' erstellt
Asset 'Brushed-Metal-1024x1024.png' erstellt
Asset 'XXXXX.png' erstellt
Asset 'XXXXX.png' erstellt
Asset 'XXXXX.tif' erstellt
Asset 'XXXXX.tif' erstellt
Asset 'MetalStainlessSteelBrushedElongated005_COL_8K_SPECULAR.jpg' erstellt
Asset 'MetalStainlessSteelBrushedElongated005_GLOSS_8K_SPECULAR.jpg' erstellt
Asset 'MetalStainlessSteelBrushedElongated005_NRM16_8K_SPECULAR.tif' erstellt
Lade Asset(s) herunter
Asset 'studio003.hdr' erstellt
2019/02/12 07:54:44  [Corona4D] [TR] Sending HELLO to server
2019/02/12 07:54:45  [Corona4D] [TR] Connection established
2019/02/12 07:54:48  [Corona4D] REPORT: unchanged: 0, Prim: 28, Inst: 0, Mtl: 0, Env-tentative: 1
2019/02/12 07:54:48  [Corona4D] Core::renderFrame before onFrame
2019/02/12 07:54:49  [Corona4D] Parsing scene took 27.69 seconds
2019/02/12 07:54:49  [Corona4D] Calculating displacement
2019/02/12 07:54:51  [Corona4D] Preparing geometry took 2.793 seconds
2019/02/12 07:54:51  [Corona4D] Preparing geometry
2019/02/12 07:54:52  [Corona4D] Embree: memory usage: 313,2 MB. Total embree build time is 537ms, top level commit took: 419ms. There are 28 embree geometry groups, 0 instances, 28/0/0/0 tri/animTri/hair/custom primitives
2019/02/12 07:54:52  [Corona4D] Core::renderFrame after onFrame
2019/02/12 07:54:52  [Corona4D] CoronaCore::renderFrame before unique materials enum
2019/02/12 07:54:52  [Corona4D] CoronaCore::renderFrame after unique materials enum
2019/02/12 07:54:52  [Corona4D] CoronaCore::renderFrame after unique materials update
2019/02/12 07:54:52  [Corona4D] Preparing geometry took 0.736 seconds
2019/02/12 07:54:52  [Corona4D] Preparing lights
2019/02/12 07:54:52  [Corona4D] TreeAdapt built: nodes 27, time 0 ms, memory 6.12 kB
2019/02/12 07:54:52  [Corona4D] CutCache initialized: records 1993496, time 29 ms, memory 79.7401 MB
2019/02/12 07:54:52  [Corona4D] CoronaCore:RenderFrame: after directLight
2019/02/12 07:54:52  [Corona4D] CoronaCore::renderFrame: after GI precompute
2019/02/12 07:54:52  [Corona4D] Preparing lights took 0.033 seconds
2019/02/12 07:54:52  [Corona4D] CoronaCore::renderFrame: before render
2019/02/12 07:54:53  [Corona4D] Rendering initial pass
2019/02/12 07:57:13  [Corona4D] [TR] Sending chunk 7/41 to the server
2019/02/12 07:59:59  [Corona4D] [TR] Sending chunk 8/41 to the server
2019/02/12 08:01:00  [Corona4D] Rendering pass 2
2019/02/12 08:02:22  [Corona4D] Rendering initial pass
2019/02/12 08:02:22  [Corona4D] [TR] Sending chunk 9/41 to the server
2019/02/12 08:04:48  [Corona4D] [TR] Sending chunk 10/41 to the server
2019/02/12 08:06:51  [Corona4D] [TR] Sending chunk 11/41 to the server
2019/02/12 08:07:11  [Corona4D] Rendering pass 2
2019/02/12 08:09:05  [Corona4D] Rendering initial pass
2019/02/12 08:09:05  [Corona4D] [TR] Sending chunk 12/41 to the server
2019/02/12 08:09:27  [Corona4D] [TR] Frame synchronization failed: Communication Error
2019/02/12 08:09:27  [Corona4D] [TR] Sending chunk 12/41 to the server
2019/02/12 08:09:28  [Corona4D] [TR] Frame synchronization failed: Communication Error
2019/02/12 08:09:28  [Corona4D] [TR] Sending chunk 12/41 to the server
2019/02/12 08:09:28  [Corona4D] [TR] Frame synchronization failed: Communication Error
2019/02/12 08:09:29  [Corona4D] [TR] Sending chunk 12/41 to the server
2019/02/12 08:09:29  [Corona4D] [TR] Frame synchronization failed: Communication Error
2019/02/12 08:09:29  [Corona4D] [TR] Sending chunk 12/41 to the server
2019/02/12 08:09:30  [Corona4D] [TR] Frame synchronization failed: Communication Error
2019/02/12 08:09:30  [Corona4D] [TR] Sending chunk 12/41 to the server
2019/02/12 08:09:30  [Corona4D] [TR] Frame synchronization failed: Communication Error
2019/02/12 08:09:31  [Corona4D] [TR] Sending chunk 12/41 to the server
2019/02/12 08:09:31  [Corona4D] [TR] Frame synchronization failed: Communication Error
2019/02/12 08:09:31  [Corona4D] CoronaCore::renderFrame: after render
2019/02/12 08:09:31  [Corona4D] Terminating DR slaves
2019/02/12 08:09:31  [Corona4D]  - terminating slave handlers
2019/02/12 08:09:31  [Corona4D]  - waiting for broadcast thread to finish
2019/02/12 08:09:31  [Corona4D]  - clearing slave handlers
2019/02/12 08:09:31  [Corona4D] Terminating DR slaves ended
2019/02/12 08:09:31  [Corona4D] Rendering took 878.941 seconds
2019/02/12 08:09:31  [Corona4D] Cleaning up
2019/02/12 08:09:31  [Corona4D] Terminating DR slaves
2019/02/12 08:09:31  [Corona4D]  - terminating slave handlers
2019/02/12 08:09:31  [Corona4D]  - waiting for broadcast thread to finish
2019/02/12 08:09:31  [Corona4D]  - clearing slave handlers
2019/02/12 08:09:31  [Corona4D] Terminating DR slaves ended
2019/02/12 08:09:31  [Corona4D] CutCache memory: 80.398 MB
2019/02/12 08:09:31  [Corona4D] Unique Primitives: 4303813
2019/02/12 08:09:31  [Corona4D] Primitives with instancing: 4303813
2019/02/12 08:09:31  [Corona4D] Area lights: 15
2019/02/12 08:09:31  [Corona4D] Geometry groups: 28
2019/02/12 08:09:31  [Corona4D] Materials: 20
2019/02/12 08:09:31  [Corona4D] Instances: 28
2019/02/12 08:09:31  [Corona4D] Portals: 0
2019/02/12 08:09:31  [Corona4D] Area lights: 15
2019/02/12 08:09:31  [Corona4D] Avg samples per pixel: 0.00105628
2019/02/12 08:09:31  [Corona4D] Avg rays per sample: 18.6773
2019/02/12 08:09:31  [Corona4D] Rays/s: 2935.24
2019/02/12 08:09:31  [Corona4D] Samples/s: 157.155
2019/02/12 08:09:31  [Corona4D] Saving + Cleaning up took 0.06 seconds
2019/02/12 08:09:31  [Corona4D] CoronaCore::exiting renderFrame
2019/02/12 08:09:31  [Corona4D] Rendered 0/0 passes
Peer-to-Peer-Statistik:
    > CAD660023 Downloadgeschwindigkeit 9.69 MiB\s (1x)
    > PC01 Downloadgeschwindigkeit 10.45 MiB\s (8x)
    > CAD660029 Downloadgeschwindigkeit 33.05 MiB\s (13x)
    > CAD660026 Downloadgeschwindigkeit 41.52 MiB\s (1x)
    > CAD660028 Downloadgeschwindigkeit 88.59 MiB\s (1x)
Peer-to-Peer-Statistik Ende

Code: [Select]
Rendering frame 0 of job 'pv'
2019/02/12 07:54:21  [Corona4D] Parsing scene
2019/02/12 07:54:21  [Corona4D] ====RENDER STARTED====
2019/02/12 07:54:21  [Corona4D] Core build timestamp: Jan 17 2019 14:36:24
Asset 'XXXXXX.psd' erstellt
Lade Asset(s) herunter
Asset 'XXXXXX.tif' erstellt
Asset 'XXXXXX.tif' erstellt
Asset 'Brushed-Metal-1024x1024.png' erstellt
Asset 'XXXXXX.png' erstellt
Asset 'XXXXXX.png' erstellt
Asset 'XXXXXX.tif' erstellt
Asset 'XXXXXX.tif' erstellt
Asset 'MetalStainlessSteelBrushedElongated005_COL_8K_SPECULAR.jpg' erstellt
Asset 'MetalStainlessSteelBrushedElongated005_GLOSS_8K_SPECULAR.jpg' erstellt
Asset 'MetalStainlessSteelBrushedElongated005_NRM16_8K_SPECULAR.tif' erstellt
Lade Asset(s) herunter
Asset 'studio003.hdr' erstellt
2019/02/12 07:54:41  [Corona4D] [TR] Sending HELLO to server
2019/02/12 07:54:41  [Corona4D] [TR] Connection established
2019/02/12 07:54:44  [Corona4D] REPORT: unchanged: 0, Prim: 28, Inst: 0, Mtl: 0, Env-tentative: 1
2019/02/12 07:54:44  [Corona4D] Core::renderFrame before onFrame
2019/02/12 07:54:45  [Corona4D] Parsing scene took 23.676 seconds
2019/02/12 07:54:45  [Corona4D] Calculating displacement
2019/02/12 07:54:48  [Corona4D] Preparing geometry took 2.863 seconds
2019/02/12 07:54:48  [Corona4D] Preparing geometry
2019/02/12 07:54:48  [Corona4D] Embree: memory usage: 313,2 MB. Total embree build time is 538ms, top level commit took: 419ms. There are 28 embree geometry groups, 0 instances, 28/0/0/0 tri/animTri/hair/custom primitives
2019/02/12 07:54:48  [Corona4D] Core::renderFrame after onFrame
2019/02/12 07:54:48  [Corona4D] CoronaCore::renderFrame before unique materials enum
2019/02/12 07:54:48  [Corona4D] CoronaCore::renderFrame after unique materials enum
2019/02/12 07:54:48  [Corona4D] CoronaCore::renderFrame after unique materials update
2019/02/12 07:54:48  [Corona4D] Preparing geometry took 0.731 seconds
2019/02/12 07:54:48  [Corona4D] Preparing lights
2019/02/12 07:54:49  [Corona4D] TreeAdapt built: nodes 27, time 0 ms, memory 6.12 kB
2019/02/12 07:54:49  [Corona4D] CutCache initialized: records 1993496, time 29 ms, memory 79.7401 MB
2019/02/12 07:54:49  [Corona4D] CoronaCore:RenderFrame: after directLight
2019/02/12 07:54:49  [Corona4D] CoronaCore::renderFrame: after GI precompute
2019/02/12 07:54:49  [Corona4D] Preparing lights took 0.033 seconds
2019/02/12 07:54:49  [Corona4D] CoronaCore::renderFrame: before render
2019/02/12 07:54:49  [Corona4D] Rendering initial pass
2019/02/12 07:57:30  [Corona4D] [TR] Sending chunk 4/41 to the server
2019/02/12 08:00:34  [Corona4D] [TR] Sending chunk 5/41 to the server
2019/02/12 08:00:58  [Corona4D] Rendering pass 2
2019/02/12 08:02:58  [Corona4D] Rendering initial pass
2019/02/12 08:02:58  [Corona4D] [TR] Sending chunk 6/41 to the server
2019/02/12 08:05:55  [Corona4D] [TR] Sending chunk 7/41 to the server
2019/02/12 08:07:09  [Corona4D] Rendering pass 2
2019/02/12 08:08:38  [Corona4D] Rendering initial pass
2019/02/12 08:08:38  [Corona4D] [TR] Sending chunk 8/41 to the server
2019/02/12 08:08:39  [Corona4D] CoronaCore::renderFrame: after render
2019/02/12 08:08:39  [Corona4D] Terminating DR slaves
2019/02/12 08:08:39  [Corona4D]  - terminating slave handlers
2019/02/12 08:08:39  [Corona4D]  - waiting for broadcast thread to finish
2019/02/12 08:08:39  [Corona4D]  - clearing slave handlers
2019/02/12 08:08:39  [Corona4D] Terminating DR slaves ended
2019/02/12 08:08:39  [Corona4D] Rendering took 830.658 seconds
2019/02/12 08:08:39  [Corona4D] Cleaning up
2019/02/12 08:08:39  [Corona4D] Terminating DR slaves
2019/02/12 08:08:39  [Corona4D]  - terminating slave handlers
2019/02/12 08:08:39  [Corona4D]  - waiting for broadcast thread to finish
2019/02/12 08:08:39  [Corona4D]  - clearing slave handlers
2019/02/12 08:08:39  [Corona4D] Terminating DR slaves ended
2019/02/12 08:08:39  [Corona4D] CutCache memory: 80.4062 MB
2019/02/12 08:08:39  [Corona4D] Unique Primitives: 4303813
2019/02/12 08:08:39  [Corona4D] Primitives with instancing: 4303813
2019/02/12 08:08:39  [Corona4D] Area lights: 15
2019/02/12 08:08:39  [Corona4D] Geometry groups: 28
2019/02/12 08:08:39  [Corona4D] Materials: 20
2019/02/12 08:08:39  [Corona4D] Instances: 28
2019/02/12 08:08:39  [Corona4D] Portals: 0
2019/02/12 08:08:39  [Corona4D] Area lights: 15
2019/02/12 08:08:39  [Corona4D] Avg samples per pixel: 0.00275053
2019/02/12 08:08:39  [Corona4D] Avg rays per sample: 18.5701
2019/02/12 08:08:39  [Corona4D] Rays/s: 8041.15
2019/02/12 08:08:39  [Corona4D] Samples/s: 433.016
2019/02/12 08:08:39  [Corona4D] Saving + Cleaning up took 0.056 seconds
2019/02/12 08:08:39  [Corona4D] CoronaCore::exiting renderFrame
2019/02/12 08:08:39  [Corona4D] Rendered 0/0 passes
2019/02/12 08:08:40  [Corona4D] [TR] Sending chunk 9/41 to the server
2019/02/12 08:08:45  [Corona4D] [TR] Finished chunks: 1/41
2019/02/12 08:08:45  [Corona4D] [TR] Sending chunk 10/41 to the server
2019/02/12 08:08:58  [Corona4D] [TR] Finished chunks: 2/41
2019/02/12 08:08:59  [Corona4D] [TR] Sending chunk 11/41 to the server
2019/02/12 08:09:08  [Corona4D] [TR] Finished chunks: 3/41
2019/02/12 08:09:08  [Corona4D] [TR] Sending chunk 12/41 to the server
2019/02/12 08:09:16  [Corona4D] [TR] Finished chunks: 4/41
2019/02/12 08:09:17  [Corona4D] [TR] Sending chunk 13/41 to the server
2019/02/12 08:09:38  [Corona4D] [TR] Frame synchronization failed: Communication Error
2019/02/12 08:09:38  [Corona4D] [TR] Sending chunk 13/41 to the server
2019/02/12 08:09:38  [Corona4D] [TR] Frame synchronization failed: Communication Error
2019/02/12 08:09:39  [Corona4D] [TR] Sending chunk 13/41 to the server
2019/02/12 08:09:39  [Corona4D] [TR] Frame synchronization failed: Communication Error
2019/02/12 08:09:39  [Corona4D] [TR] Sending chunk 13/41 to the server
2019/02/12 08:09:39  [Corona4D] [TR] Frame synchronization failed: Communication Error
2019/02/12 08:09:40  [Corona4D] [TR] Sending chunk 13/41 to the server
2019/02/12 08:09:40  [Corona4D] [TR] Frame synchronization failed: Communication Error
2019/02/12 08:09:40  [Corona4D] [TR] Sending chunk 13/41 to the server
2019/02/12 08:09:41  [Corona4D] [TR] Frame synchronization failed: Communication Error
2019/02/12 08:09:41  [Corona4D] [TR] Sending chunk 13/41 to the server
2019/02/12 08:09:41  [Corona4D] [TR] Frame synchronization failed: Communication Error
Peer-to-Peer-Statistik:
    > CAD660024 Downloadgeschwindigkeit 4.38 MiB\s (1x)
    > CAD660026 Downloadgeschwindigkeit 9.15 MiB\s (1x)
    > PC01 Downloadgeschwindigkeit 12.24 MiB\s (7x)
    > CAD660029 Downloadgeschwindigkeit 38.18 MiB\s (13x)
    > CAD660027 Downloadgeschwindigkeit 50.73 MiB\s (1x)
    > CAD660025 Downloadgeschwindigkeit 53.85 MiB\s (1x)
Peer-to-Peer-Statistik Ende

And here is one example of a machine without errors:

Code: [Select]
Rendering frame 0 of job 'pv'
2019/02/12 07:54:22  [Corona4D] Parsing scene
2019/02/12 07:54:22  [Corona4D] ====RENDER STARTED====
2019/02/12 07:54:22  [Corona4D] Core build timestamp: Jan 17 2019 14:36:24
Asset 'XXXXXX.psd' erstellt
Lade Asset(s) herunter
Asset 'XXXXXX.tif' erstellt
Asset 'XXXXXX.tif' erstellt
Asset 'Brushed-Metal-1024x1024.png' erstellt
Asset 'XXXXXX.png' erstellt
Asset 'XXXXXX.png' erstellt
Asset 'XXXXXX.tif' erstellt
Asset 'XXXXXX.tif' erstellt
Asset 'MetalStainlessSteelBrushedElongated005_COL_8K_SPECULAR.jpg' erstellt
Asset 'MetalStainlessSteelBrushedElongated005_GLOSS_8K_SPECULAR.jpg' erstellt
Asset 'MetalStainlessSteelBrushedElongated005_NRM16_8K_SPECULAR.tif' erstellt
Asset 'studio003.hdr' erstellt
2019/02/12 07:54:42  [Corona4D] [TR] Sending HELLO to server
2019/02/12 07:54:42  [Corona4D] [TR] Connection established
2019/02/12 07:54:45  [Corona4D] REPORT: unchanged: 0, Prim: 28, Inst: 0, Mtl: 0, Env-tentative: 1
2019/02/12 07:54:45  [Corona4D] Core::renderFrame before onFrame
2019/02/12 07:54:45  [Corona4D] Parsing scene took 23.556 seconds
2019/02/12 07:54:45  [Corona4D] Calculating displacement
2019/02/12 07:54:48  [Corona4D] Preparing geometry took 2.851 seconds
2019/02/12 07:54:48  [Corona4D] Preparing geometry
2019/02/12 07:54:49  [Corona4D] Embree: memory usage: 315,2 MB. Total embree build time is 532ms, top level commit took: 414ms. There are 28 embree geometry groups, 0 instances, 28/0/0/0 tri/animTri/hair/custom primitives
2019/02/12 07:54:49  [Corona4D] Core::renderFrame after onFrame
2019/02/12 07:54:49  [Corona4D] CoronaCore::renderFrame before unique materials enum
2019/02/12 07:54:49  [Corona4D] CoronaCore::renderFrame after unique materials enum
2019/02/12 07:54:49  [Corona4D] CoronaCore::renderFrame after unique materials update
2019/02/12 07:54:49  [Corona4D] Preparing geometry took 0.726 seconds
2019/02/12 07:54:49  [Corona4D] Preparing lights
2019/02/12 07:54:49  [Corona4D] TreeAdapt built: nodes 27, time 0 ms, memory 6.12 kB
2019/02/12 07:54:49  [Corona4D] CutCache initialized: records 1993496, time 29 ms, memory 79.7401 MB
2019/02/12 07:54:49  [Corona4D] CoronaCore:RenderFrame: after directLight
2019/02/12 07:54:49  [Corona4D] CoronaCore::renderFrame: after GI precompute
2019/02/12 07:54:49  [Corona4D] Preparing lights took 0.033 seconds
2019/02/12 07:54:49  [Corona4D] CoronaCore::renderFrame: before render
2019/02/12 07:54:50  [Corona4D] Rendering initial pass
2019/02/12 07:57:29  [Corona4D] [TR] Sending chunk 3/41 to the server
2019/02/12 08:00:32  [Corona4D] [TR] Sending chunk 4/41 to the server
2019/02/12 08:00:53  [Corona4D] Rendering pass 2
2019/02/12 08:03:24  [Corona4D] Rendering initial pass
2019/02/12 08:03:24  [Corona4D] [TR] Sending chunk 5/41 to the server
2019/02/12 08:06:22  [Corona4D] [TR] Sending chunk 6/41 to the server
2019/02/12 08:06:58  [Corona4D] Rendering pass 2
2019/02/12 08:08:17  [Corona4D] Rendering initial pass
2019/02/12 08:08:17  [Corona4D] [TR] Sending chunk 7/41 to the server
2019/02/12 08:08:20  [Corona4D] CoronaCore::renderFrame: after render
2019/02/12 08:08:20  [Corona4D] Terminating DR slaves
2019/02/12 08:08:20  [Corona4D]  - terminating slave handlers
2019/02/12 08:08:20  [Corona4D]  - waiting for broadcast thread to finish
2019/02/12 08:08:20  [Corona4D]  - clearing slave handlers
2019/02/12 08:08:20  [Corona4D] Terminating DR slaves ended
2019/02/12 08:08:20  [Corona4D] Rendering took 810.947 seconds
2019/02/12 08:08:20  [Corona4D] Cleaning up
2019/02/12 08:08:20  [Corona4D] Terminating DR slaves
2019/02/12 08:08:20  [Corona4D]  - terminating slave handlers
2019/02/12 08:08:20  [Corona4D]  - waiting for broadcast thread to finish
2019/02/12 08:08:20  [Corona4D]  - clearing slave handlers
2019/02/12 08:08:20  [Corona4D] Terminating DR slaves ended
2019/02/12 08:08:20  [Corona4D] CutCache memory: 80.3913 MB
2019/02/12 08:08:20  [Corona4D] Unique Primitives: 4303813
2019/02/12 08:08:20  [Corona4D] Primitives with instancing: 4303813
2019/02/12 08:08:20  [Corona4D] Area lights: 15
2019/02/12 08:08:20  [Corona4D] Geometry groups: 28
2019/02/12 08:08:20  [Corona4D] Materials: 20
2019/02/12 08:08:20  [Corona4D] Instances: 28
2019/02/12 08:08:20  [Corona4D] Portals: 0
2019/02/12 08:08:20  [Corona4D] Area lights: 15
2019/02/12 08:08:20  [Corona4D] Avg samples per pixel: 0.0077253
2019/02/12 08:08:20  [Corona4D] Avg rays per sample: 18.5262
2019/02/12 08:08:20  [Corona4D] Rays/s: 23079.2
2019/02/12 08:08:20  [Corona4D] Samples/s: 1245.76
2019/02/12 08:08:20  [Corona4D] Saving + Cleaning up took 0.066 seconds
2019/02/12 08:08:20  [Corona4D] CoronaCore::exiting renderFrame
2019/02/12 08:08:20  [Corona4D] Rendered 0/0 passes
2019/02/12 08:08:21  [Corona4D] [TR] Sending chunk 8/41 to the server
2019/02/12 08:08:22  [Corona4D] [TR] Finished chunks: 1/41
2019/02/12 08:08:23  [Corona4D] [TR] Sending chunk 9/41 to the server
2019/02/12 08:08:25  [Corona4D] [TR] Finished chunks: 2/41
2019/02/12 08:08:25  [Corona4D] [TR] Sending chunk 10/41 to the server
2019/02/12 08:08:29  [Corona4D] [TR] Finished chunks: 3/41
2019/02/12 08:08:29  [Corona4D] [TR] Sending chunk 11/41 to the server
2019/02/12 08:08:49  [Corona4D] [TR] Finished chunks: 4/41
2019/02/12 08:08:49  [Corona4D] [TR] Sending chunk 12/41 to the server
2019/02/12 08:09:00  [Corona4D] [TR] Finished chunks: 5/41
2019/02/12 08:09:01  [Corona4D] [TR] Sending chunk 13/41 to the server
2019/02/12 08:09:12  [Corona4D] [TR] Finished chunks: 6/41
2019/02/12 08:09:12  [Corona4D] [TR] Sending chunk 14/41 to the server
2019/02/12 08:09:22  [Corona4D] [TR] Finished chunks: 7/41
2019/02/12 08:09:22  [Corona4D] [TR] Sending chunk 15/41 to the server
2019/02/12 08:09:27  [Corona4D] [TR] Finished chunks: 8/41
2019/02/12 08:09:28  [Corona4D] [TR] Sending chunk 16/41 to the server
2019/02/12 08:09:31  [Corona4D] [TR] Finished chunks: 9/41
2019/02/12 08:09:32  [Corona4D] [TR] Sending chunk 17/41 to the server
2019/02/12 08:09:36  [Corona4D] [TR] Finished chunks: 10/41
2019/02/12 08:09:37  [Corona4D] [TR] Sending chunk 18/41 to the server
2019/02/12 08:09:44  [Corona4D] [TR] Finished chunks: 11/41
2019/02/12 08:09:45  [Corona4D] [TR] Sending chunk 19/41 to the server
2019/02/12 08:09:50  [Corona4D] [TR] Finished chunks: 12/41
2019/02/12 08:09:51  [Corona4D] [TR] Sending chunk 20/41 to the server
2019/02/12 08:10:04  [Corona4D] [TR] Finished chunks: 13/41
2019/02/12 08:10:04  [Corona4D] [TR] Sending chunk 21/41 to the server
2019/02/12 08:10:07  [Corona4D] [TR] Finished chunks: 14/41
2019/02/12 08:10:08  [Corona4D] [TR] Sending chunk 22/41 to the server
2019/02/12 08:10:14  [Corona4D] [TR] Finished chunks: 15/41
2019/02/12 08:10:15  [Corona4D] [TR] Sending chunk 23/41 to the server
2019/02/12 08:10:19  [Corona4D] [TR] Finished chunks: 16/41
2019/02/12 08:10:19  [Corona4D] [TR] Sending chunk 24/41 to the server
2019/02/12 08:10:25  [Corona4D] [TR] Finished chunks: 17/41
2019/02/12 08:10:26  [Corona4D] [TR] Sending chunk 25/41 to the server
2019/02/12 08:10:30  [Corona4D] [TR] Finished chunks: 18/41
2019/02/12 08:10:31  [Corona4D] [TR] Sending chunk 26/41 to the server
2019/02/12 08:10:40  [Corona4D] [TR] Finished chunks: 19/41
2019/02/12 08:10:41  [Corona4D] [TR] Sending chunk 27/41 to the server
2019/02/12 08:10:44  [Corona4D] [TR] Finished chunks: 20/41
2019/02/12 08:10:45  [Corona4D] [TR] Sending chunk 28/41 to the server
2019/02/12 08:10:48  [Corona4D] [TR] Finished chunks: 21/41
2019/02/12 08:10:49  [Corona4D] [TR] Sending chunk 29/41 to the server
2019/02/12 08:10:51  [Corona4D] [TR] Finished chunks: 22/41
2019/02/12 08:10:52  [Corona4D] [TR] Sending chunk 30/41 to the server
2019/02/12 08:10:54  [Corona4D] [TR] Finished chunks: 23/41
2019/02/12 08:10:55  [Corona4D] [TR] Sending chunk 31/41 to the server
2019/02/12 08:11:02  [Corona4D] [TR] Finished chunks: 24/41
2019/02/12 08:11:02  [Corona4D] [TR] Sending chunk 32/41 to the server
2019/02/12 08:11:05  [Corona4D] [TR] Finished chunks: 25/41
2019/02/12 08:11:05  [Corona4D] [TR] Sending chunk 33/41 to the server
2019/02/12 08:11:08  [Corona4D] [TR] Finished chunks: 26/41
2019/02/12 08:11:08  [Corona4D] [TR] Sending chunk 34/41 to the server
2019/02/12 08:11:11  [Corona4D] [TR] Finished chunks: 27/41
2019/02/12 08:11:11  [Corona4D] [TR] Sending chunk 35/41 to the server
2019/02/12 08:11:13  [Corona4D] [TR] Finished chunks: 28/41
2019/02/12 08:11:13  [Corona4D] [TR] Sending chunk 36/41 to the server
2019/02/12 08:11:14  [Corona4D] [TR] Finished chunks: 29/41
2019/02/12 08:11:15  [Corona4D] [TR] Sending chunk 37/41 to the server
2019/02/12 08:11:16  [Corona4D] [TR] Finished chunks: 30/41
2019/02/12 08:11:16  [Corona4D] [TR] Sending chunk 38/41 to the server
2019/02/12 08:11:22  [Corona4D] [TR] Finished chunks: 31/41
2019/02/12 08:11:23  [Corona4D] [TR] Sending chunk 39/41 to the server
2019/02/12 08:11:25  [Corona4D] [TR] Finished chunks: 32/41
2019/02/12 08:11:25  [Corona4D] [TR] Sending chunk 40/41 to the server
2019/02/12 08:11:27  [Corona4D] [TR] Finished chunks: 33/41
2019/02/12 08:11:27  [Corona4D] [TR] Sending chunk 0/41 to the server
2019/02/12 08:11:31  [Corona4D] [TR] Finished chunks: 34/41
2019/02/12 08:11:32  [Corona4D] [TR] Sending chunk 1/41 to the server
2019/02/12 08:11:34  [Corona4D] [TR] Finished chunks: 35/41
2019/02/12 08:11:34  [Corona4D] [TR] Sending chunk 2/41 to the server
2019/02/12 08:11:35  [Corona4D] [TR] Finished chunks: 36/41
2019/02/12 08:11:36  [Corona4D] [TR] Sending chunk 3/41 to the server
2019/02/12 08:11:39  [Corona4D] [TR] Finished chunks: 37/41
2019/02/12 08:11:40  [Corona4D] [TR] Sending chunk 4/41 to the server
2019/02/12 08:11:41  [Corona4D] [TR] Finished chunks: 38/41
2019/02/12 08:11:42  [Corona4D] [TR] Sending chunk 5/41 to the server
2019/02/12 08:11:45  [Corona4D] [TR] Finished chunks: 39/41
2019/02/12 08:11:46  [Corona4D] [TR] Sending chunk 6/41 to the server
2019/02/12 08:11:49  [Corona4D] [TR] Finished chunks: 40/41
2019/02/12 08:11:50  [Corona4D] [TR] Sending chunk 7/41 to the server
2019/02/12 08:11:51  [Corona4D] [TR] Finished chunks: 41/41
Peer-to-Peer-Statistik:
    > CAD660022 Downloadgeschwindigkeit 10.09 MiB\s (1x)
    > PC01 Downloadgeschwindigkeit 10.62 MiB\s (8x)
    > CAD660029 Downloadgeschwindigkeit 34.58 MiB\s (13x)
    > CAD660023 Downloadgeschwindigkeit 41.25 MiB\s (1x)
    > CAD660025 Downloadgeschwindigkeit 48.63 MiB\s (1x)
Peer-to-Peer-Statistik Ende

I only censored some asset-names.

Hope that helps!

Greetings,
Moritz

2019-02-12, 09:07:22
Reply #38

HFPatzi

  • Active Users
  • **
  • Posts: 92
    • View Profile
Ok.....one last thing :D

I rendered the whole scene with Physical Renderer and TR for comparison.

Rendertime: 24min, no dropouts.
Here is one example client console-log:

Code: [Select]
Grabbed Render-Job 'pv' from machine 'PC01'
Asset 'XXXXXX.c4d' erstellt
Lade Asset(s) herunter
Lade Asset(s) herunter
Lade Asset(s) herunter
Downloaded Asset(s) in 9.771 seconds
Start Rendering for Machine PC01
Asset 'XXXXXX.psd' erstellt
Lade Asset(s) herunter
Asset 'XXXXXX.tif' erstellt
Asset 'XXXXXX.tif' erstellt
Asset 'Brushed-Metal-1024x1024.png' erstellt
Asset 'XXXXXX.png' erstellt
Asset 'XXXXXX.tif' erstellt
Asset 'XXXXXX.tif' erstellt
Asset 'MetalStainlessSteelBrushedElongated005_GLOSS_8K_SPECULAR.jpg' erstellt
Asset 'MetalStainlessSteelBrushedElongated005_NRM16_8K_SPECULAR.tif' erstellt
Asset 'studio003.hdr' erstellt
Rendering frame 0 of job 'pv'
Peer-to-Peer-Statistik:
    > CAD660027 Downloadgeschwindigkeit 4.84 MiB\s (1x)
    > PC01 Downloadgeschwindigkeit 22.14 MiB\s (9x)
    > CAD660026 Downloadgeschwindigkeit 30.18 MiB\s (1x)
    > CAD660028 Downloadgeschwindigkeit 35.49 MiB\s (10x)
    > CAD660024 Downloadgeschwindigkeit 77.72 MiB\s (1x)
Peer-to-Peer-Statistik Ende

2019-02-12, 10:15:27
Reply #39

kizo

  • Active Users
  • **
  • Posts: 28
    • View Profile
Hello Moritz,

its great to see you are testing this too. Seems the tests show the same pattern where the faster the render gets the more clients drop out.Makes me think how great and fast would it be if none would disconnect.
Getting no disconnects but a rather slow render time is rather frustrating when you have more power available.

"The main Problem i have with TR right now is, that one has to do a lot of tests to find out the perfect render settings/ number of clients for every project. Almost all time advantage you have with TR is eaten up by finding the best settings for each project"

yes I can agree thats a big problem for me too. It is scene and resolution dependent but also the number of nodes used plays a role so to optimally use the clients one should do extensive testing before rendering and thats taking way too much time

@Tom

I did test a scene from the content browser "Living Room.c4d"  and I got clients disconnecting.I managed to do only one test using all clients for now but will try to do more and let you know.

Cheers

kizo

2019-02-12, 14:38:11
Reply #40

TomG

  • Corona Team
  • Active Users
  • ****
  • Posts: 2737
    • View Profile
Wow, thanks guys for the tests, very much appreciated!

Moritz, what was the render limit, was it noise or passes? Any chance you made a note of how many passes were done? (One thing to note about distributed rendering, and this applies to Max too, is that it may "go over" the render limit, e.g. target is 4% noise, master receives updates of 30 passes from nodes and actual noise is 4.1% so it continues rendering, then when it hits 4% if receives another 30 passes from the clients and the noise gets to 3% - so it can mean you end up with more passes and a cleaner image than a single workstation, which can obscure some time savings just a little; the more nodes are in use, the more likely that is to happen by a larger margin - though it still shouldn't be massive, but it may play a part).

Interesting that the Win machines seem to drop out more than the Mac machines. (EDIT though there are more Win clients, so a roll of the dice on who drops out makes it more likely to be a Win machine... so maybe there isn't a difference or a big difference, as I read more :) )

Also interesting Kizo that a native render engine and scene also seems to show the drop outs, could point to just a general flaw in TR itself rather than how we implement it (not to say that there isn't anything we can do if that's the case, we'd certainly still look into it, but it is just good to know where to look and what solutions to consider and investigate).

Thanks!
« Last Edit: 2019-02-12, 14:45:22 by TomG »

2019-02-12, 15:08:12
Reply #41

HFPatzi

  • Active Users
  • **
  • Posts: 92
    • View Profile
You're right, i forgot to mention that :) On all tests, pass limit was set to15 and no denoising activated.
I also noticed the "overshoot" when pass limit is set. e.g. 26 of 15 passes.
I will test the "Living Room.c4d" scene too asap and will report the results

2019-02-12, 16:13:03
Reply #42

HFPatzi

  • Active Users
  • **
  • Posts: 92
    • View Profile
Any chance you made a note of how many passes were done?

You mean how many more passes then the 15 it was set up for? If so, I haven't unfortunatly. But i can tell that it was always more then 15 (maybe between 5 and 15 more) and not alway the same.

2019-02-12, 16:19:03
Reply #43

TomG

  • Corona Team
  • Active Users
  • ****
  • Posts: 2737
    • View Profile
NP on not having the exact number, good to know what the rendering limit was though, and yes that expected "overrun" will account for some time differences (render time may not reduce quite as much as expected, but that's because there are actually more passes being done / a less noisy render being produced).

2019-02-16, 09:30:54
Reply #44

Gnorkie

  • Active Users
  • **
  • Posts: 21
    • View Profile
Same issues here in the company I work for.

We already bought 7 ws+5nodes licenses and a ws+10 nodes license.

We could potentially use 50 clients to render, and yes... 1gbit Ethernet.

I tried last night a 5000px A3 image, quite simple file, and clients were constantly getting disconnected and I was manually restarting them from the main workstation.

Sorry guys at Corona but this is quite disappointing since you are selling this product and not every company has time to do so many tests like Kizo is doing here, some companies might not even read the forum.

As written in one of the posts I definitely suggest you to develop a dedicated DR server. VRAYforC4D's one was (and is) working smoothly, using the vrayserver from Chaos. Now Chaos acquired them, let's see what comes.
The company I work for was using Maxwell Render before switching to Corona (we are trying to...) and we never had such issues, even using 20 or 30 nodes to calculate a single image.

I might sound harsh but you are selling a product that you didn't develop personally and also it seems you didn't test properly. I'm not talking about Corona but about the Team Render solution.


EDIT:

Reading the other post I see people complaining for the vrayserver restrictions. Well, the later a proper DR will be developed the more users will feel the "loss" of native C4D shaders usability.
« Last Edit: 2019-02-16, 09:35:53 by Gnorkie »