NSS DB migration problem

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

NSS DB migration problem

Hans Petter Jansson-2
I've been looking in NSS docs and ML archives for a solution to the
following problem, with no luck. Asking around on IRC, I was pointed to
this ML. Hopefully it's the appropriate forum.

I'm trying to migrate existing NSS DBs from the old Berkeley DB format
to the new, SQLite-based format that can be shared by multiple
applications, using a snipped of C code from your migration guide. The
relevant code looks like this (abbreviated for clarity):

  /* STEP 1: Signal that update/merge may be needed  */

  rv = NSS_InitWithMerge ("sql:/home/user/new_db",
                          "", "", "secmod.db",
                          "/home/user/old_db", "", "",
                          "myapp", "myapp",
                          flags);

  if (rv == SECFailure)
  {
    /* Fallback code */
    return rv;
  }

  slot = PK11_GetInternalKeySlot ();

  /* Step 2: Determine if update/merge is needed. */

  if (PK11_IsRemovable (slot))
  {
    /* need to update/Merge the database */

    /* Step 3: Authenticate to the token */

    rv = PK11_Authenticate (slot, PR_FALSE, &pwdata);
    if (rv != SECSuccess)
    {
      fprintf (stderr, "*** Auth call failed: %u.\n", PORT_GetError ());
      return rv;
    }
  }

This works for some databases, but not others. It doesn't seem to matter
which application created the database (I've tried with databases from
Firefox and Evolution) - e.g. one user's database may fail while another
user's database may migrate properly. When it fails, it's always on the
first PK11_Authenticate () call (step 3). The code above produces the
following output:

*** Auth call failed: 4294959104.

That is, 0xffffe000. If I set up an auth callback, it never gets called.
Do you have any suggestions as to what I'm doing wrong here?

--
Hans Petter Jansson <[hidden email]>

_______________________________________________
dev-tech-crypto mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-crypto
Reply | Threaded
Open this post in threaded view
|

Re: NSS DB migration problem

Wan-Teh Chang-3
On Fri, Nov 14, 2008 at 9:54 PM, Hans Petter Jansson <[hidden email]> wrote:

>
>    /* Step 3: Authenticate to the token */
>
>    rv = PK11_Authenticate (slot, PR_FALSE, &pwdata);
>    if (rv != SECSuccess)
>    {
>      fprintf (stderr, "*** Auth call failed: %u.\n", PORT_GetError ());
>      return rv;
>    }
>  }
>
> This works for some databases, but not others. It doesn't seem to matter
> which application created the database (I've tried with databases from
> Firefox and Evolution) - e.g. one user's database may fail while another
> user's database may migrate properly. When it fails, it's always on the
> first PK11_Authenticate () call (step 3). The code above produces the
> following output:
>
> *** Auth call failed: 4294959104.
>
> That is, 0xffffe000. If I set up an auth callback, it never gets called.
> Do you have any suggestions as to what I'm doing wrong here?

Bob Relyea is the best person to answer your question.  I just
wanted to suggest a change to your code.  PORT_GetError()
returns 'int', so you should print it with the %d format.

The error code you get is -8192, which is SEC_ERROR_IO:
http://www.mozilla.org/projects/security/pki/nss/ref/ssl/sslerr.html

The fact that PK11_Authenticate failed, the error code is
SEC_ERROR_IO, and your auth callback never gets called implies
that the PK11_NeedUserInit call in PK11_DoPassword failed:
http://mxr.mozilla.org/security/source/security/nss/lib/pk11wrap/pk11auth.c#553

553     if (PK11_NeedUserInit(slot)) {
554         PORT_SetError(SEC_ERROR_IO);
555         return SECFailure;
556     }

This is all I can fine out...

Bob, SEC_ERROR_IO seems like a strange error code for reporting
that PK11_NeedUserInit(slot) failed.

Wan-Teh
_______________________________________________
dev-tech-crypto mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-crypto
Reply | Threaded
Open this post in threaded view
|

Re: NSS DB migration problem

Nelson B Bolyard-2
In reply to this post by Hans Petter Jansson-2
Hans Petter Jansson wrote, On 2008-11-14 21:54:
> I've been looking in NSS docs and ML archives for a solution to the
> following problem, with no luck. Asking around on IRC, I was pointed to
> this ML. Hopefully it's the appropriate forum.
>
> I'm trying to migrate existing NSS DBs from the old Berkeley DB format
> to the new, SQLite-based format that can be shared by multiple
> applications, using a snipped of C code from your migration guide.

[code snipped]

> This works for some databases, but not others. It doesn't seem to matter
> which application created the database (I've tried with databases from
> Firefox and Evolution) - e.g. one user's database may fail while another
> user's database may migrate properly.

Do these separate users have separate systems, which separate installations
of the software?  Or do they share a single multi-user system?

I'm wondering if perhaps differences in their installations can explain it.

> When it fails, it's always on the first PK11_Authenticate () call (step
> 3). The code above produces the following output:
>
> *** Auth call failed: 4294959104.
>
> That is, 0xffffe000.

which as a signed number is -8192, which is SEC_ERROR_IO.

http://www.mozilla.org/projects/security/pki/nss/ref/ssl/sslerr.html#1039257

> If I set up an auth callback, it never gets called.

SEC_ERROR_IO was originally intended to have a specific meaning, which
was something like "I wanted to ask you for your password, but I couldn't
open the file or device (terminal) with which to get it, so I gave up."
That meaning would be consistent with a failure in PK11_Authenticate.

However, it has been overloaded with another meaning which is much more
vague, basically "We tried to do something with a PKCS#11 token and it
failed".  For one example of that, see
https://bugzilla.mozilla.org/show_bug.cgi?id=464411

> Do you have any suggestions as to what I'm doing wrong here?

Let's start with basics.
Might some of them have corrupt cert8 DBs?
Can you get a listing of their cert DBs with certutil?

Does modutil show any unloaded PKCS#11 modules in their secmod DB?

What version of NSS are you using?
This sounds somewhat like
https://bugzilla.mozilla.org/show_bug.cgi?id=397122
but that was fixed a year ago.
_______________________________________________
dev-tech-crypto mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-crypto
Reply | Threaded
Open this post in threaded view
|

Re: NSS DB migration problem

Hans Petter Jansson-2
On Fri, 2008-11-14 at 22:56 -0800, Nelson B Bolyard wrote:
> Hans Petter Jansson wrote, On 2008-11-14 21:54:

> > This works for some databases, but not others. It doesn't seem to matter
> > which application created the database (I've tried with databases from
> > Firefox and Evolution) - e.g. one user's database may fail while another
> > user's database may migrate properly.

> Do these separate users have separate systems, which separate installations
> of the software?  Or do they share a single multi-user system?
>
> I'm wondering if perhaps differences in their installations can explain it.

It's on separate workstations, but in some cases one database migrates
successfully while another fails on the same system. In one case, I have
a Firefox database that migrates successfully (resulting in a 150KB
migrated database, slightly smaller than the source) and a small
Evolution database that does not migrate, for the same user on the same
workstation with the same software versions.

> which as a signed number is -8192, which is SEC_ERROR_IO.

Ah yes, sorry. I was using %d initially, then changed it to %u while
looking for the problem and subsequently forgot about it :)

> Let's start with basics.
> Might some of them have corrupt cert8 DBs?

I doubt it - this affects many users on different systems, and the old
databases are working in their original contexts. Could there be an
entry in the database that cannot be migrated, or that confuses the
migration code for some reason?

> Can you get a listing of their cert DBs with certutil?

Here's one:

test@test:~/.mozilla/firefox/lim69e3n.default> certutil -L -d .

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

VeriSign Class 3 Secure Server CA                            ,,
Go Daddy Secure Certification Authority                      ,,
Cybertrust Educational CA                                    ,,
Deutsche Telekom CA 5                                        ,,
campusonline.uni-freiburg.de                                 ,,
NewMedia-NET GmbH                                            ,,
DPWN Root CA R2 PS                                           ,,
GlobalSign                                                   ,,
XRamp Security Services GS CA                                ,,
GlobalSign RootSign Partners CA                              ,,
VeriSign, Inc.                                               ,,
VeriSign Class 3 Extended Validation SSL SGC CA              ,,
VeriSign Class 3 Public Primary Certification Authority - G5 ,,
imtek.uni-freiburg.de                                        ,,
I.T. Telecom Global CA                                       ,,
DPWN SSL CA I2 PS                                            ,,
GlobalSign Extended Validation CA                            ,,

And another:

´╗┐test@test:~/.mozilla/firefox/lim69e3n.default> certutil -L -d .              

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

Microsoft Secure Server Authority                            ,,  
VeriSign Class 3 Extended Validation SSL CA                  ,,  
GlobalSign                                                   ,,  
Deutsche Telekom CA 5                                        ,,  
www.asensys.de                                               ,,  
EssentialSSL CA                                              ,,
Microsoft Internet Authority                                 ,,
VeriSign Class 3 Extended Validation SSL SGC CA              ,,
VeriSign, Inc.                                               ,,
WebSpace-Forum Server CA                                     ,,
UTN-USERFirst-Hardware                                       ,,
Akamai Subordinate CA 3                                      ,,
Starfield Secure Certification Authority                     ,,
GlobalSign Extended Validation CA                            ,,
GlobalSign RootSign Partners CA                              ,,
Thawte SGC CA                                                ,,
VeriSign Class 3 Secure Server CA                            ,,
Go Daddy Secure Certification Authority                      ,,
cfc-comm.cfc-int                                             ,,
DigiCert Global CA                                           ,,

> Does modutil show any unloaded PKCS#11 modules in their secmod DB?

No. They tend to look like this:

test@test:~/.mozilla/firefox/lim69e3n.default> modutil -dbdir . -list

Listing of PKCS #11 Modules
-----------------------------------------------------------
  1. NSS Internal PKCS #11 Module
         slots: 2 slots attached
        status: loaded

         slot: NSS Internal Cryptographic Services
        token: NSS Generic Crypto Services

         slot: NSS User Private Key and Certificate Services
        token: NSS Certificate DB
-----------------------------------------------------------

Although for one database that consistently fails to migrate, it looks
like this:

test@test:~/.evolution> modutil -dbdir . -list

Listing of PKCS #11 Modules
-----------------------------------------------------------
  1. NSS Internal PKCS #11 Module
         slots: 2 slots attached
        status: loaded

         slot: NSS Internal Cryptographic Services
        token: NSS Generic Crypto Services

         slot: NSS User Private Key and Certificate Services
        token: NSS Certificate DB

  2. Mozilla Root Certs
        library name: /usr/lib/libnssckbi.so
         slots: 1 slot attached
        status: loaded

         slot: NSS Builtin Objects
        token: Builtin Object Token
-----------------------------------------------------------

This database only fails to migrate if the target database was not
already created by another, successful merge, though.

> What version of NSS are you using?

3.12.0.

> This sounds somewhat like
> https://bugzilla.mozilla.org/show_bug.cgi?id=397122
> but that was fixed a year ago.

Yeah... I think I can provide you with a problem DB in private that you
can look at if you've got the time.

--
Hans Petter Jansson <[hidden email]>

_______________________________________________
dev-tech-crypto mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-crypto
Reply | Threaded
Open this post in threaded view
|

Re: NSS DB migration problem

Nelson B Bolyard-2
Hans Petter Jansson wrote, On 2008-11-15 17:57:
> On Fri, 2008-11-14 at 22:56 -0800, Nelson B Bolyard wrote:
>> Hans Petter Jansson wrote, On 2008-11-14 21:54:
>
>>> This works for some databases, but not others.

> It's on separate workstations, but in some cases one database migrates
> successfully while another fails on the same system.

Is is possible that more than one version of the NSS shared libraries is
installed, and that the two users have different LD_LIBRARY_PATHs, so
that the two users are getting different versions of the NSS when they
attempt this?

>> Let's start with basics.
>> Might some of them have corrupt cert8 DBs?
>
> I doubt it - this affects many users on different systems, and the old
> databases are working in their original contexts. Could there be an
> entry in the database that cannot be migrated, or that confuses the
> migration code for some reason?

Perhaps.  The migration tries to copy ALL the old DB entries.  It's possible
to have a bogus entry that never gets used by the browser in
normal circumstances, and so causes no problem for the browser, but is
a problem for the migration software.

The fact that you can get DB listings with certutil is evidence that
suggests that the DB is not corrupted, but I wouldn't call it disproof.


> This database only fails to migrate if the target database was not
> already created by another, successful merge, though.

I think you're saying that the failures only occur if the "target" (cert9)
DB doesn't already exist when your program is run, but does succeed if
the target is already a legit cert9 DB.  Is that what you're saying?
That would not surprise me.  I think that the function you're using,
NSS_InitWithMerge, does assume that the target DB already exists.

>> What version of NSS are you using?
>
> 3.12.0.

At this point, I think the problem probably lies in one of these classes
of problems (in no particular order or probability):

a) The DB is corrupt
b) Using a "wrong" version of the NSS shared libs
c) a bug in NSS 3.12.0
d) a bug in your program

I can suggest a couple more tests that might help eliminate one or more
of those classes.

With the browser not running, and the cert8 DBs not in use, and the DBs
in a directory that has no cert9.db or key4.db, use certutil from NSS 3.12.x
to do this command:
   certutil -L -X -d sql:DBDIR
where "sql:" is that literal prefix (that's lower case SQL), and
DBDIR is the path name of the directory that holds the cert8.db file.
If that succeeds, it should generate cert9 and key4.db files in the
directory that held the cert8.  You can then try the command
   certutil -L -d sql:DBDIR       (no -X this time)
and if that also lists the certs, then the conversion was probably
successful.  That would pretty strongly disprove a DB corruption.
OTOH, if that also fails, it suggests there is a DB problem.

Note that this test won't be using NSS_InitWithMerge.  It's using
ordinary NSS_Init.  It won't merge two DBs together.  It only upgrades
the old DB.  However, knowing if that works, or not, will help us
narrow down the problem, and decide what to do next.

> Yeah... I think I can provide you with a problem DB in private that you
> can look at if you've got the time.

Let's try some other things first.

/Nelson
_______________________________________________
dev-tech-crypto mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-crypto
Reply | Threaded
Open this post in threaded view
|

Re: NSS DB migration problem

Robert Relyea
In reply to this post by Hans Petter Jansson-2
Hans Petter Jansson wrote:

>
> This works for some databases, but not others. It doesn't seem to matter
> which application created the database (I've tried with databases from
> Firefox and Evolution) - e.g. one user's database may fail while another
> user's database may migrate properly. When it fails, it's always on the
> first PK11_Authenticate () call (step 3). The code above produces the
> following output:
>
> *** Auth call failed: 4294959104.
>
> That is, 0xffffe000. If I set up an auth callback, it never gets called.
> Do you have any suggestions as to what I'm doing wrong here?
>  
Do these databases have passwords set on them?

Also, does certutil --merge on these databases?

bob


_______________________________________________
dev-tech-crypto mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-crypto

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: NSS DB migration problem

Hans Petter Jansson-2
In reply to this post by Nelson B Bolyard-2
On Sat, 2008-11-15 at 19:06 -0800, Nelson B Bolyard wrote:
> Hans Petter Jansson wrote, On 2008-11-15 17:57:

> > It's on separate workstations, but in some cases one database migrates
> > successfully while another fails on the same system.

> Is is possible that more than one version of the NSS shared libraries is
> installed, and that the two users have different LD_LIBRARY_PATHs, so
> that the two users are getting different versions of the NSS when they
> attempt this?

It's highly unlikely.

> > This database only fails to migrate if the target database was not
> > already created by another, successful merge, though.

> I think you're saying that the failures only occur if the "target" (cert9)
> DB doesn't already exist when your program is run, but does succeed if
> the target is already a legit cert9 DB.  Is that what you're saying?
> That would not surprise me.  I think that the function you're using,
> NSS_InitWithMerge, does assume that the target DB already exists.

Almost. I'm saying that if the database fails to migrate the first time,
and leaves an empty database, then subsequent merges to that database
also fail. However, if the first merge is successful (creating the
database and populating it), then subsequent merges always work.

I have a small sample size, though, since I don't have (nor want,
really) direct access to other users' databases.

It looks like Wolfgang found the problem, or at least a problem, in the
"Firefox' password manager" thread, so I'm going to test using a normal
NSS_Init() to create the database if it doesn't already exist, and see
if that works.

Thanks, Wolfgang :)

--
Hans Petter Jansson <[hidden email]>

_______________________________________________
dev-tech-crypto mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-crypto
Reply | Threaded
Open this post in threaded view
|

Re: NSS DB migration problem

Wolfgang Rosenauer-2
In reply to this post by Nelson B Bolyard-2
Hi,

Hans Petter Jansson schrieb:

>>> This database only fails to migrate if the target database was not
>>> already created by another, successful merge, though.
>
>> I think you're saying that the failures only occur if the "target" (cert9)
>> DB doesn't already exist when your program is run, but does succeed if
>> the target is already a legit cert9 DB.  Is that what you're saying?
>> That would not surprise me.  I think that the function you're using,
>> NSS_InitWithMerge, does assume that the target DB already exists.
>
> Almost. I'm saying that if the database fails to migrate the first time,
> and leaves an empty database, then subsequent merges to that database
> also fail. However, if the first merge is successful (creating the
> database and populating it), then subsequent merges always work.
>
> I have a small sample size, though, since I don't have (nor want,
> really) direct access to other users' databases.
>
> It looks like Wolfgang found the problem, or at least a problem, in the
> "Firefox' password manager" thread, so I'm going to test using a normal
> NSS_Init() to create the database if it doesn't already exist, and see
> if that works.
>
> Thanks, Wolfgang :)

I _think_ the problem when creating a new database (if one doesn't
already exist) using NSS_Init() could be that the internal token (SDR)
is created and people will never be able to read their passwords again
even after a merge? I think what should be done is to mimic exactly what
"certutil -L -X -d sql:." is doing for the initial creation of the database?

Wolfgang
_______________________________________________
dev-tech-crypto mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-crypto
Reply | Threaded
Open this post in threaded view
|

Re: NSS DB migration problem

Robert Relyea
Wolfgang Rosenauer wrote:

> Hi,
>
> Hans Petter Jansson schrieb:
>  
>>>> This database only fails to migrate if the target database was not
>>>> already created by another, successful merge, though.
>>>>        
>>> I think you're saying that the failures only occur if the "target" (cert9)
>>> DB doesn't already exist when your program is run, but does succeed if
>>> the target is already a legit cert9 DB.  Is that what you're saying?
>>> That would not surprise me.  I think that the function you're using,
>>> NSS_InitWithMerge, does assume that the target DB already exists.
>>>      
>> Almost. I'm saying that if the database fails to migrate the first time,
>> and leaves an empty database, then subsequent merges to that database
>> also fail. However, if the first merge is successful (creating the
>> database and populating it), then subsequent merges always work.
>>
>> I have a small sample size, though, since I don't have (nor want,
>> really) direct access to other users' databases.
>>
>> It looks like Wolfgang found the problem, or at least a problem, in the
>> "Firefox' password manager" thread, so I'm going to test using a normal
>> NSS_Init() to create the database if it doesn't already exist, and see
>> if that works.
>>
>> Thanks, Wolfgang :)
>>    
>
> I _think_ the problem when creating a new database (if one doesn't
> already exist) using NSS_Init() could be that the internal token (SDR)
> is created and people will never be able to read their passwords again
> even after a merge? I think what should be done is to mimic exactly what
> "certutil -L -X -d sql:." is doing for the initial creation of the database?
>  
In the single cert8/key3 update to cert9/key4 (as opposed to a merge
update where you are merging from cert8/key3 to a cert9/key4 in a shared
location) case, which is what we are talking about here, there is a 2
step update. The second step requires a password. The code is supposed
to know if the update has only half completed (I had some problem with
early version of the shared DB code), where the databases are created,
but the tables have not been. When it comes time to do the actual
migrate, the tables are then created. If this is the case that's
failing, we can simulate it with certutil as follows:

Start it a dbm database with a password.

certutil -L -X -d sql:.

This command does not require a password, so the databases will only
partially update (databases are created, but not populated).

certutil -K -x -d sql:.
<supply the password>

This should successfully update the database.
I've just tried this and it failed. So this looks like a good test to
reproduce the problem. I'll attach it to the bug.

bob

> Wolfgang
> _______________________________________________
> dev-tech-crypto mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-tech-crypto
>  


_______________________________________________
dev-tech-crypto mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-crypto

smime.p7s (4K) Download Attachment