Mac notarization fails about 1/4 of the time: "A timestamp was expected but was not found"

[Tom Goddard]

Our nightly builds on the Mac often fail because of a notarization failure when running the Mac codesign command. The error message is "A timestamp was expected but was not found". Our notarization code retries the codesign command but often it fails again.

Others have reported the same problem and found it to be firewalls sporadically blocking packets from Apple's timestamp server

​https://developer.apple.com/forums/thread/8187

The suggested solution is to get your network IT people to fix this, or to move you notarization to a network that doesn't have this firewall issue.

[Tom Goddard]

Our use of codesign specifies the "--timestamp" option. The codesign man page says about this option

During signing, requests that a timestamp authority server be contacted to authenticate the time of signing. The server contacted is given by the URL value.  If this option is given without a value, a default server provided by Apple is used.  Note that this server may not support signatures made with identities not furnished by Apple.  If the timestamp authority service cannot be  contacted over the Internet, or it malfunctions or refuses service, the signing operation will fail. If this option is not given at all, a system-specific default behavior is invoked.  This may result in some but not all code signatures being timestamped. The special value none explicitly disables the use of timestamp services.

[Tom Goddard]

The notarization code runs /usr/bin/codesign on shared libraries and executables in 239 directories. Directories have anywhere from one to about 30 binaries that get signed. But codesign fails frequently with message "A timestamp was expected but was not found." For example in the August 9 Intel Mac daily build it failed 9 times on different directories. Our notarization code will retry running codesign on each directory if it fails. If it fails twice for the same directory notarization fails and the build is not notarized and is not put on the download page. The August 9 daily build had 9 codesign failures but all succeeded on the second try so the build was notarized. The Mac ARM techpreview build on August 9 had codesign fail twice on the same directory (python3.9/lib-dynload) and so it was not notarized and no Mac ARM build was put on the download page.

Here is an example from the daily build Mac output where codesign failed and then succeeded on the second try.

signing ['chimerax/ChimeraX_Daily.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib\
/python3.9/site-packages/chimerax/mlp/_mlp.cpython-39-darwin.so'] failed:                           
command: ['/usr/bin/codesign', '--keychain', '/Users/chimera/Library/Keychains/codesign.keychain', \
'--sign', '(LWV8X224YF)', '--options=runtime', '--timestamp', '--entitlements=macosx_package/entitl\
ements.plist', '--force', '--strict', '--verbose=4', 'chimerax/ChimeraX_Daily.app/Contents/Library/\
Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/chimerax/mlp/_mlp.cpython-39-d\
arwin.so']                                                                                          
stderr:                                                                                             
chimerax/ChimeraX_Daily.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9\
/site-packages/chimerax/mlp/_mlp.cpython-39-darwin.so: A timestamp was expected but was not found.  
                                                                                                    
retry codesign: ['chimerax/ChimeraX_Daily.app/Contents/Library/Frameworks/Python.framework/Versions\
/3.9/lib/python3.9/site-packages/chimerax/mlp/_mlp.cpython-39-darwin.so']                           
signing ['chimerax/ChimeraX_Daily.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib\
/python3.9/site-packages/chimerax/mlp/_mlp.cpython-39-darwin.so'] succeeded                         
signing ['chimerax/ChimeraX_Daily.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib\
/python3.9/site-packages/cftime/_cftime.cpython-39-darwin.so'] succeeded                            
stderr:                                                                                             
chimerax/ChimeraX_Daily.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9\
/site-packages/cftime/_cftime.cpython-39-darwin.so: signed Mach-O thin (x86_64) [_cftime.cpython-39\
-darwin] 

There are hundreds of reports online about codesign failing in this way. Apple usually claims this is a developer firewall problem. But the failures are often sporadic. Still it might be the firewall blocking after too many requests in a short interval, possibly looking like a denial of service attack. The build process sleeps for 10 seconds after a failure before attempting to codesign again.

Since the failures seem sporadic I have increased the number of tries to 5 and the sleep time to 30 seconds after the first failure, 60 seconds for the second failure, 90 seconds after the third failure ... to see if this reduces notarization failures. This change is in the chimerax/build_tools git repository on plato in macosx_package/notarize.py.

[Tom Goddard]

The motivation for trying to improve notarization reliability is that we plan to make a Mac universal build. That will require Intel, ARM and universal builds to all notarize successfully in order that the universal daily build gets to the download page, and that could fail half the time our current notarization failure rates.

[Tom Goddard]

I increased the number of notarization retries from 1 to 4 and failures are now very rare. We are notarizing 9 mac builds every night (arm, intel and universal, daily, parallel, and candidate release) the past 6 weeks and notarization has failed a total of maybe 5 times, so failing less than 1 in 50 times. This is good enough. Hopefully Apple will improve the notarization robustness some year.