NSS - TLS Session Hash and Extended Master Secret doesn't work in FIPS mode with Java 1.8.0_161

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

NSS - TLS Session Hash and Extended Master Secret doesn't work in FIPS mode with Java 1.8.0_161

james.h.covington
Not sure if this post belongs here or in mozilla.dev.tech.crypto, have posted to both:

I am using nss 3.23.0.0 with nspr 4.12.0.0 in FIPS mode to provide the cryptography for my Red Hat JBoss EAP 6.4.14 application server hosted on Windows 2008 R2 server. After updating my JDK version from 1.8.0_152 to 1.8.0_162, I received this exception during TLS handshake in my server.log:

  RSA master secret generation error:
 java.security.InvalidAlgorithmParameterException: Key format must be RAW
  at com.sun.crypto.provider.TlsMasterSecretGenerator.engineInit(TlsMasterSecretGenerator.java:67)
  at javax.crypto.KeyGenerator.init(KeyGenerator.java:454)
  at javax.crypto.KeyGenerator.init(KeyGenerator.java:430)
  at sun.security.ssl.Handshaker.calculateMasterSecret(Handshaker.java:1261)
  at sun.security.ssl.Handshaker.calculateKeys(Handshaker.java:1183)
  at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:301)
  at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
  at sun.security.ssl.Handshaker.process_record(Handshaker.java:987)
  at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1072)
  at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
  at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
  at sun.security.ssl.SSLSocketImpl.getSession(SSLSocketImpl.java:2306)
  at org.apache.tomcat.util.net.jsse.JSSESocketFactory.handshake(JSSESocketFactory.java:201)
  at org.apache.tomcat.util.net.JIoEndpoint.setSocketOptions(JIoEndpoint.java:1114)
  at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926)
  at java.lang.Thread.run(Thread.java:748)

In reviewing Oracle's release notes, I found that one of the new features effective in build 161 is the addition of TLS Session Hash and Extended Master Secret extension support in the jsse provider. By following the guidance in the release note, and adding -Djdk.tls.UseExtendedMasterSecret=false to my JAVA_OPTS variable, the exception is no longer raised.

I have read that NSS supports Extended Master Secret extension effective with version 3.21, so I'm not sure where to direct a bug report. Perhaps this extension is not supported in FIPS mode?
_______________________________________________
dev-security mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-security
Reply | Threaded
Open this post in threaded view
|

Re: NSS - TLS Session Hash and Extended Master Secret doesn't work in FIPS mode with Java 1.8.0_161

james.h.covington
Doing more reading on Extended Master Secret extension support in NSS, I understand that support is "off" by default. But I can't see in any documentation how to "turn it on". Any help out there?

On Monday, February 26, 2018 at 8:57:16 AM UTC-5, James Covington wrote:

> Not sure if this post belongs here or in mozilla.dev.tech.crypto, have posted to both:
>
> I am using nss 3.23.0.0 with nspr 4.12.0.0 in FIPS mode to provide the cryptography for my Red Hat JBoss EAP 6.4.14 application server hosted on Windows 2008 R2 server. After updating my JDK version from 1.8.0_152 to 1.8.0_162, I received this exception during TLS handshake in my server.log:
>
>   RSA master secret generation error:
>  java.security.InvalidAlgorithmParameterException: Key format must be RAW
>   at com.sun.crypto.provider.TlsMasterSecretGenerator.engineInit(TlsMasterSecretGenerator.java:67)
>   at javax.crypto.KeyGenerator.init(KeyGenerator.java:454)
>   at javax.crypto.KeyGenerator.init(KeyGenerator.java:430)
>   at sun.security.ssl.Handshaker.calculateMasterSecret(Handshaker.java:1261)
>   at sun.security.ssl.Handshaker.calculateKeys(Handshaker.java:1183)
>   at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:301)
>   at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
>   at sun.security.ssl.Handshaker.process_record(Handshaker.java:987)
>   at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1072)
>   at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
>   at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
>   at sun.security.ssl.SSLSocketImpl.getSession(SSLSocketImpl.java:2306)
>   at org.apache.tomcat.util.net.jsse.JSSESocketFactory.handshake(JSSESocketFactory.java:201)
>   at org.apache.tomcat.util.net.JIoEndpoint.setSocketOptions(JIoEndpoint.java:1114)
>   at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926)
>   at java.lang.Thread.run(Thread.java:748)
>
> In reviewing Oracle's release notes, I found that one of the new features effective in build 161 is the addition of TLS Session Hash and Extended Master Secret extension support in the jsse provider. By following the guidance in the release note, and adding -Djdk.tls.UseExtendedMasterSecret=false to my JAVA_OPTS variable, the exception is no longer raised.
>
> I have read that NSS supports Extended Master Secret extension effective with version 3.21, so I'm not sure where to direct a bug report. Perhaps this extension is not supported in FIPS mode?

_______________________________________________
dev-security mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-security
Reply | Threaded
Open this post in threaded view
|

Re: NSS - TLS Session Hash and Extended Master Secret doesn't work in FIPS mode with Java 1.8.0_161

Martin Thomson
SSL_OptionSet or SSL_OptionSetDefault with
SSL_ENABLE_EXTENDED_MASTER_SECRET and PR_TRUE.  I'm not sure how that
manifests on the Java side of the house though.

On Tue, Feb 27, 2018 at 11:59 PM, James Covington
<[hidden email]> wrote:

> Doing more reading on Extended Master Secret extension support in NSS, I understand that support is "off" by default. But I can't see in any documentation how to "turn it on". Any help out there?
>
> On Monday, February 26, 2018 at 8:57:16 AM UTC-5, James Covington wrote:
>> Not sure if this post belongs here or in mozilla.dev.tech.crypto, have posted to both:
>>
>> I am using nss 3.23.0.0 with nspr 4.12.0.0 in FIPS mode to provide the cryptography for my Red Hat JBoss EAP 6.4.14 application server hosted on Windows 2008 R2 server. After updating my JDK version from 1.8.0_152 to 1.8.0_162, I received this exception during TLS handshake in my server.log:
>>
>>   RSA master secret generation error:
>>  java.security.InvalidAlgorithmParameterException: Key format must be RAW
>>       at com.sun.crypto.provider.TlsMasterSecretGenerator.engineInit(TlsMasterSecretGenerator.java:67)
>>       at javax.crypto.KeyGenerator.init(KeyGenerator.java:454)
>>       at javax.crypto.KeyGenerator.init(KeyGenerator.java:430)
>>       at sun.security.ssl.Handshaker.calculateMasterSecret(Handshaker.java:1261)
>>       at sun.security.ssl.Handshaker.calculateKeys(Handshaker.java:1183)
>>       at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:301)
>>       at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
>>       at sun.security.ssl.Handshaker.process_record(Handshaker.java:987)
>>       at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1072)
>>       at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
>>       at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
>>       at sun.security.ssl.SSLSocketImpl.getSession(SSLSocketImpl.java:2306)
>>       at org.apache.tomcat.util.net.jsse.JSSESocketFactory.handshake(JSSESocketFactory.java:201)
>>       at org.apache.tomcat.util.net.JIoEndpoint.setSocketOptions(JIoEndpoint.java:1114)
>>       at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926)
>>       at java.lang.Thread.run(Thread.java:748)
>>
>> In reviewing Oracle's release notes, I found that one of the new features effective in build 161 is the addition of TLS Session Hash and Extended Master Secret extension support in the jsse provider. By following the guidance in the release note, and adding -Djdk.tls.UseExtendedMasterSecret=false to my JAVA_OPTS variable, the exception is no longer raised.
>>
>> I have read that NSS supports Extended Master Secret extension effective with version 3.21, so I'm not sure where to direct a bug report. Perhaps this extension is not supported in FIPS mode?
>
> _______________________________________________
> dev-security mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-security
_______________________________________________
dev-security mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-security
Reply | Threaded
Open this post in threaded view
|

Re: NSS - TLS Session Hash and Extended Master Secret doesn't work in FIPS mode with Java 1.8.0_161

Martin Thomson
Oh, I just noticed.  From your stack trace, you aren't hitting NSS
libssl, that looks like the Java implementation of TLS.

On Wed, Feb 28, 2018 at 1:53 PM, Martin Thomson <[hidden email]> wrote:

> SSL_OptionSet or SSL_OptionSetDefault with
> SSL_ENABLE_EXTENDED_MASTER_SECRET and PR_TRUE.  I'm not sure how that
> manifests on the Java side of the house though.
>
> On Tue, Feb 27, 2018 at 11:59 PM, James Covington
> <[hidden email]> wrote:
>> Doing more reading on Extended Master Secret extension support in NSS, I understand that support is "off" by default. But I can't see in any documentation how to "turn it on". Any help out there?
>>
>> On Monday, February 26, 2018 at 8:57:16 AM UTC-5, James Covington wrote:
>>> Not sure if this post belongs here or in mozilla.dev.tech.crypto, have posted to both:
>>>
>>> I am using nss 3.23.0.0 with nspr 4.12.0.0 in FIPS mode to provide the cryptography for my Red Hat JBoss EAP 6.4.14 application server hosted on Windows 2008 R2 server. After updating my JDK version from 1.8.0_152 to 1.8.0_162, I received this exception during TLS handshake in my server.log:
>>>
>>>   RSA master secret generation error:
>>>  java.security.InvalidAlgorithmParameterException: Key format must be RAW
>>>       at com.sun.crypto.provider.TlsMasterSecretGenerator.engineInit(TlsMasterSecretGenerator.java:67)
>>>       at javax.crypto.KeyGenerator.init(KeyGenerator.java:454)
>>>       at javax.crypto.KeyGenerator.init(KeyGenerator.java:430)
>>>       at sun.security.ssl.Handshaker.calculateMasterSecret(Handshaker.java:1261)
>>>       at sun.security.ssl.Handshaker.calculateKeys(Handshaker.java:1183)
>>>       at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:301)
>>>       at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
>>>       at sun.security.ssl.Handshaker.process_record(Handshaker.java:987)
>>>       at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1072)
>>>       at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
>>>       at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
>>>       at sun.security.ssl.SSLSocketImpl.getSession(SSLSocketImpl.java:2306)
>>>       at org.apache.tomcat.util.net.jsse.JSSESocketFactory.handshake(JSSESocketFactory.java:201)
>>>       at org.apache.tomcat.util.net.JIoEndpoint.setSocketOptions(JIoEndpoint.java:1114)
>>>       at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926)
>>>       at java.lang.Thread.run(Thread.java:748)
>>>
>>> In reviewing Oracle's release notes, I found that one of the new features effective in build 161 is the addition of TLS Session Hash and Extended Master Secret extension support in the jsse provider. By following the guidance in the release note, and adding -Djdk.tls.UseExtendedMasterSecret=false to my JAVA_OPTS variable, the exception is no longer raised.
>>>
>>> I have read that NSS supports Extended Master Secret extension effective with version 3.21, so I'm not sure where to direct a bug report. Perhaps this extension is not supported in FIPS mode?
>>
>> _______________________________________________
>> dev-security mailing list
>> [hidden email]
>> https://lists.mozilla.org/listinfo/dev-security
_______________________________________________
dev-security mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-security
Reply | Threaded
Open this post in threaded view
|

Re: NSS - TLS Session Hash and Extended Master Secret doesn't work in FIPS mode with Java 1.8.0_161

james.h.covington
In reply to this post by Martin Thomson
Thanks for the reply, just to clarify the configuration, I'm using NSS as described in the JDK 8 PKCS#11 Reference Guide

https://docs.oracle.com/javase/8/docs/technotes/guides/security/p11guide.html#NSS


On Tuesday, February 27, 2018 at 9:54:47 PM UTC-5, Martin Thomson wrote:

> Oh, I just noticed.  From your stack trace, you aren't hitting NSS
> libssl, that looks like the Java implementation of TLS.
>
> On Wed, Feb 28, 2018 at 1:53 PM, Martin Thomson <[hidden email]> wrote:
> > SSL_OptionSet or SSL_OptionSetDefault with
> > SSL_ENABLE_EXTENDED_MASTER_SECRET and PR_TRUE.  I'm not sure how that
> > manifests on the Java side of the house though.
> >
> > On Tue, Feb 27, 2018 at 11:59 PM, James Covington
> > <[hidden email]> wrote:
> >> Doing more reading on Extended Master Secret extension support in NSS, I understand that support is "off" by default. But I can't see in any documentation how to "turn it on". Any help out there?
> >>
> >> On Monday, February 26, 2018 at 8:57:16 AM UTC-5, James Covington wrote:
> >>> Not sure if this post belongs here or in mozilla.dev.tech.crypto, have posted to both:
> >>>
> >>> I am using nss 3.23.0.0 with nspr 4.12.0.0 in FIPS mode to provide the cryptography for my Red Hat JBoss EAP 6.4.14 application server hosted on Windows 2008 R2 server. After updating my JDK version from 1.8.0_152 to 1.8.0_162, I received this exception during TLS handshake in my server.log:
> >>>
> >>>   RSA master secret generation error:
> >>>  java.security.InvalidAlgorithmParameterException: Key format must be RAW
> >>>       at com.sun.crypto.provider.TlsMasterSecretGenerator.engineInit(TlsMasterSecretGenerator.java:67)
> >>>       at javax.crypto.KeyGenerator.init(KeyGenerator.java:454)
> >>>       at javax.crypto.KeyGenerator.init(KeyGenerator.java:430)
> >>>       at sun.security.ssl.Handshaker.calculateMasterSecret(Handshaker.java:1261)
> >>>       at sun.security.ssl.Handshaker.calculateKeys(Handshaker.java:1183)
> >>>       at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:301)
> >>>       at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
> >>>       at sun.security.ssl.Handshaker.process_record(Handshaker.java:987)
> >>>       at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1072)
> >>>       at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
> >>>       at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
> >>>       at sun.security.ssl.SSLSocketImpl.getSession(SSLSocketImpl.java:2306)
> >>>       at org.apache.tomcat.util.net.jsse.JSSESocketFactory.handshake(JSSESocketFactory.java:201)
> >>>       at org.apache.tomcat.util.net.JIoEndpoint.setSocketOptions(JIoEndpoint.java:1114)
> >>>       at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926)
> >>>       at java.lang.Thread.run(Thread.java:748)
> >>>
> >>> In reviewing Oracle's release notes, I found that one of the new features effective in build 161 is the addition of TLS Session Hash and Extended Master Secret extension support in the jsse provider. By following the guidance in the release note, and adding -Djdk.tls.UseExtendedMasterSecret=false to my JAVA_OPTS variable, the exception is no longer raised.
> >>>
> >>> I have read that NSS supports Extended Master Secret extension effective with version 3.21, so I'm not sure where to direct a bug report. Perhaps this extension is not supported in FIPS mode?
> >>
> >> _______________________________________________
> >> dev-security mailing list
> >> [hidden email]
> >> https://lists.mozilla.org/listinfo/dev-security

_______________________________________________
dev-security mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-security