Author Topic: Corona saving layered EXR on render end  (Read 22459 times)

2019-08-02, 04:27:07

Dan Rodgers

  • Active Users
  • **
  • Posts: 52
    • View Profile
Hi,

We have an issue where in some scenes, and apparently for no reason, Corona starts to save out HUGE layered exr files rather than normal single image files.

The scenes haven't really changed, a couple of geometry changes, but all render settings are the same.  We are saving as half float exr.

Does Corona even support saving as multilayered exr?  I have never seen an option for it.


This is a massive issue for us, it takes 30 mins to save out the files and because it is doing this for every render element we can easily chew through a couple of hundred Gb of space overnight.

We are using Backburner, but we have the exact same issue if we manually save straight from the VFB


Cheers

2019-08-02, 10:29:30
Reply #1

jms.lwly

  • Active Users
  • **
  • Posts: 71
    • View Profile
    • jms.lwly studio
I’ve had a similar (unexplained) issue recently - rendered on two machines via backburner, one saved out multilayered single EXRs, the other saved each channel as a separate file.

Literally the same render job from Backburner, but this happened consistently over several different camera passes.

2019-08-02, 10:53:54
Reply #2

Frood

  • Active Users
  • **
  • Posts: 1414
    • View Profile
    • Rakete GmbH
the other saved each channel as a separate file.

The other saved additionally each channel, right? The main output file still contained all other passes? It's a very strange behaviour. Sometimes I had the feeling that a node from which the job was submitted writes the (layererd) EXRs correctly (and only this) when included in the BB job. but I'm not sure. That mess in the output directories is really annoying :-|


Good Luck




Never underestimate the power of a well placed level one spell.

2019-08-02, 14:52:29
Reply #3

alexyork

  • Active Users
  • **
  • Posts: 521
  • Partner at Recent Spaces
    • View Profile
    • RECENT SPACES
Same here... happened once or twice. Unfortunately no time to diagnose.
Alex York
Partner
RECENT SPACES
recentspaces.com

2019-08-02, 15:16:12
Reply #4

Frood

  • Active Users
  • **
  • Posts: 1414
    • View Profile
    • Rakete GmbH
no time to diagnose.

Don't waste your time, because I fear you will find nothing. This is -again- a "since ever" issue with max :(

Does Corona even support saving as multilayered exr?  I have never seen an option for it.

If you use EXR as output and choose the "Setup" button you can activate "Automatically Add/Remove Render Elements from Render Dialog". This would put all your render elements into one EXR (Max feature, not a Corona feature). In pratice it happens often that additionally the render elements also get saved as extra files. And CXR (which is basically multilayered EXR + additional Corona specific information) has it all inclusive anyway :)


Good Luck



Never underestimate the power of a well placed level one spell.

2019-08-02, 16:11:37
Reply #5

alexyork

  • Active Users
  • **
  • Posts: 521
  • Partner at Recent Spaces
    • View Profile
    • RECENT SPACES
Are you guys using Deadline always? Maybe related?
Alex York
Partner
RECENT SPACES
recentspaces.com

2019-08-12, 06:11:56
Reply #6

Dan Rodgers

  • Active Users
  • **
  • Posts: 52
    • View Profile
Cheers for all the info.  Never realised there was a native layered option within the 3ds max setup dialogue. It makes me surer that this is probably just a 3ds max issue, no wonder VRay had their own save options from years ago..

We are only using Backburner here.

We constantly get this issue though (see attached), where the files aren't even layered but are all huge and identical sizes except the beauty.  I can't split them with exr-io or in Fusion/AE  and it gobbles up server space on a crazy level.  Only seems to do it on large renders, 5k and above

2019-08-12, 09:07:31
Reply #7

Frood

  • Active Users
  • **
  • Posts: 1414
    • View Profile
    • Rakete GmbH
Oh, this looks like you are not using any compression. Go to "setup" while you are defining the output and use zip compression or wavelet (yes, the latter is also lossless, no worries).


Good Luck



Never underestimate the power of a well placed level one spell.

2019-08-14, 12:37:24
Reply #8

maru

  • Corona Team
  • Active Users
  • ****
  • Posts: 9443
  • Marcin
    • View Profile
Is the below correct?

- You would like to save only a single EXR file with all render elements included in it
- Right now, instead of a single file, you are getting the "main" EXR + all other render elements saved separately as EXR files

---

One extra note:
If you use the "Save CXR" option and then convert your CXR to EXR, you will end up with a large file containing some layers which are normally hidden (for example information used to perform denoising). Maybe you are getting this kind of files? Could you send us some samples? The uploader is in my signature.

2019-08-14, 19:04:22
Reply #9

Frood

  • Active Users
  • **
  • Posts: 1414
    • View Profile
    • Rakete GmbH
Right now, instead of a single file, you are getting the "main" EXR + all other render elements saved separately as EXR files

I guess it's again a Max issue not passing the file type properties (does not depend on output type EXR or CXR, even not on the renderer). We noticed that the node from which the job was submitted does it correctly if included in the BB job. Meaning: it saves only that one multilayered exr in this instance just like intended. All others do whatever. Completely weird, but also there "forever" in Max. Additionally I guess it's about the local settings of a node, just like JPG quality (had this issue recently while scripting).

Just tried it: seems to be as guessed :)

Use attached script as prerender script for a BB job (Render Setup -> Scripts -> Pre-Prender) and look at the max.log of the participating nodes: on nodes which write all additional passes you get 'false'. On nodes which work as intended you get 'true'. The script also sets this value to 'true' so that if you render ONE job with the script as prerender script on all nodes, the issue should be solved because the setting is locally stored on the node. Time to add this as another prerender script network wide.


Good Luck


Edit: This affects EXR only unfortunately, but not CXR.
Edit: Removed the script to avoid data loss. Reason: When using it (permanently as prerender script) and not wanting one layered EXR as output but multiple ones, additional REs are not saved at all. We always do layered EXRs so it does what it should do for us.
« Last Edit: 2019-09-19, 17:02:26 by Frood »
Never underestimate the power of a well placed level one spell.

2019-08-23, 03:34:03
Reply #10

Dan Rodgers

  • Active Users
  • **
  • Posts: 52
    • View Profile
Oh, this looks like you are not using any compression. Go to "setup" while you are defining the output and use zip compression or wavelet (yes, the latter is also lossless, no worries).


Good Luck

That's how it is set up anyway, with ZIP compression.

Is the below correct?

- You would like to save only a single EXR file with all render elements included in it
- Right now, instead of a single file, you are getting the "main" EXR + all other render elements saved separately as EXR files

---

One extra note:
If you use the "Save CXR" option and then convert your CXR to EXR, you will end up with a large file containing some layers which are normally hidden (for example information used to perform denoising). Maybe you are getting this kind of files? Could you send us some samples? The uploader is in my signature.

No.  I want all my exr's saved out separate (which is happening).  The issue is that the files are bigger than they should be. MUCH bigger. Even if I make a mistake and have an empty render element i.e Volumetrics but I have no volumes in the scene, it will save out a massive file with no info in it.  Check the image above, there is no way that my alpha should be more than double the size of my beauty pass, and it has no extra info in there that I can access.



So after more testing, this isn't just an issue with Backburner...  and we dont get the issue with VRay, Redshift etc.  It seems to happen with only the high res files.  3k, we get the expected different file sizes per pass, jump to 6k or 7k and we get all [asses with the same file size.  This also happens with TIFFs.  And across every machine in the studio.


I genuinely think this is a Corona issue, something about saving large-resolution files seems to make it shit it's pants, the contents of which are filling up our server space

2019-08-23, 09:45:53
Reply #11

rowmanns

  • Corona Team
  • Active Users
  • ****
  • Posts: 603
    • View Profile
Hi,

Does this only happen when using backburner? Or does it also happen when rendering with Corona on its own?

Secondly, would you be able to provide a scene where this is happening for you? And I will do my best to investigate the issue.

Instructions on how to send the scene to us are in my signiture.

Thanks,

Rowan
Please read this before reporting bugs:
How to report issues to us!

2019-08-25, 13:53:27
Reply #12

valwizard

  • Active Users
  • **
  • Posts: 55
  • "We get up in the morning. We do our best. Nothing
    • View Profile
It is happening when using DR. I have created a topic but perhaps it was deleted in order to not duplicate the matter
"We get up in the morning. We do our best. Nothing else matters! "

2019-09-02, 08:08:36
Reply #13

Dan Rodgers

  • Active Users
  • **
  • Posts: 52
    • View Profile
Hi,

Does this only happen when using backburner? Or does it also happen when rendering with Corona on its own?

Secondly, would you be able to provide a scene where this is happening for you? And I will do my best to investigate the issue.

Instructions on how to send the scene to us are in my signiture.

Thanks,

Rowan

this happens with both.  The only way to get 'normal sized' render passes is to save as CXR then save out p[asses individually from that.

I'll try and get a file over to you by then end of the week

2019-09-02, 16:30:02
Reply #14

rowmanns

  • Corona Team
  • Active Users
  • ****
  • Posts: 603
    • View Profile
Thanks, I'll look forward to receiving the scene.
Please read this before reporting bugs:
How to report issues to us!