urlunsplit for itms-services scheme returns invalid url

Relating to a Werkzueg issue (pallets/werkzeug#2691), when parsing an iOS App install url e.g.
itms-services:action=download-manifest&url=https://theacmeinc.com/abcdefeg, urlunpslit returns an invalid url.

e.g.

vals = urlparse( "itms-services://?action=download-manifest&url=https://theacmeinc.com/abcdefeg" )
print(vals)

newURL = urlunsplit((vals.scheme, vals.netloc, vals.path, vals.query, vals.params))
print( newURL )

prints:

ParseResult(scheme='itms-services', netloc='', path='', params='', query='action=download-manifest&url=https://theacmeinc.com/abcdefeg', fragment='')

itms-services:?action=download-manifest&url=https://theacmeinc.com/abcdefeg

Note the newURL is missing the // after the itms-services scheme.

This scheme is used to install ad-hoc and enterprise iOS apps.

Your environment

Tested on Apple M1 Max - 13.4 Beta (22F5049e)
Python: 3.10.10

For more details on the scheme here is a link to the Apple documentation (look for the "Use a website to distribute the app" section).
https://support.apple.com/en-gb/guide/deployment/depce7cefc4d/web

Linked PRs

gh-104139: Add itms-services to uses_netloc urllib.parse. #104312

I don't think this scheme is standard. https://url.spec.whatwg.org/#url-serializing says "If url's host is non-null: Append "//" to output." This itms-services URL doesn't have a host, so it shouldn't require a //. I think there's some other special casing for schemes with the uses_netloc list, a similar "add //" could be added.

cpython/Lib/urllib/parse.py

Lines 44 to 74 in 9f9e001

    
           # A classification of schemes. 
        
           # The empty string classifies URLs with no scheme specified, 
        
           # being the default value returned by “urlsplit” and “urlparse”. 
        
           uses_relative = ['', 'ftp', 'http', 'gopher', 'nntp', 'imap', 
        
                            'wais', 'file', 'https', 'shttp', 'mms', 
        
                            'prospero', 'rtsp', 'rtspu', 'sftp', 
        
                            'svn', 'svn+ssh', 'ws', 'wss'] 
        
           uses_netloc = ['', 'ftp', 'http', 'gopher', 'nntp', 'telnet', 
        
                          'imap', 'wais', 'file', 'mms', 'https', 'shttp', 
        
                          'snews', 'prospero', 'rtsp', 'rtspu', 'rsync', 
        
                          'svn', 'svn+ssh', 'sftp', 'nfs', 'git', 'git+ssh', 
        
                          'ws', 'wss'] 
        
           uses_params = ['', 'ftp', 'hdl', 'prospero', 'http', 'imap', 
        
                          'https', 'shttp', 'rtsp', 'rtspu', 'sip', 'sips', 
        
                          'mms', 'sftp', 'tel'] 
        
           # These are not actually used anymore, but should stay for backwards 
        
           # compatibility.  (They are undocumented, but have a public-looking name.) 
        
           non_hierarchical = ['gopher', 'hdl', 'mailto', 'news', 
        
                               'telnet', 'wais', 'imap', 'snews', 'sip', 'sips'] 
        
           uses_query = ['', 'http', 'wais', 'imap', 'https', 'shttp', 'mms', 
        
                         'gopher', 'rtsp', 'rtspu', 'sip', 'sips'] 
        
           uses_fragment = ['', 'ftp', 'hdl', 'http', 'gopher', 'news', 
        
                            'nntp', 'wais', 'https', 'shttp', 'snews', 
        
                            'file', 'prospero']

@gpshead

@gpshead I know you were looking at some other urllib issues recently, could you comment on this?

@davidism

I agree with @davidism that the WhatWG URL spec does not require // when there is no pathname.

Regardless, behavior wise this seems to match our existing uses_netloc special case so we can just add it to that list in Lib/urllib/parse.py and add test coverage in Lib/test/test_urlparse.py. I made a PR.

Workaround for this to "work" on existing Pythons:

if "itms-services" not in urllib.parse.uses_netloc:
    urllib.parse.uses_netloc.append("itms-services")

I'm calling this a feature as code will have to deal with Python's that do not list it as such for a long time anyways via a code snippet like that, so backporting doesn't seem consistently helpful.

gh-104139: Add itms-services to uses_netloc urllib.parse. (#104312)

fixed in 3.12.

Merge branch 'main' into fixsendfamily

Merge branch 'main' into supermon

Merge branch 'main' into fixloadsuperspec

Navigation Menu

Repository navigation

urlunsplit for itms-services scheme returns invalid url #104139

Your environment

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	# A classification of schemes.
	# The empty string classifies URLs with no scheme specified,
	# being the default value returned by “urlsplit” and “urlparse”.

	uses_relative = ['', 'ftp', 'http', 'gopher', 'nntp', 'imap',
	'wais', 'file', 'https', 'shttp', 'mms',
	'prospero', 'rtsp', 'rtspu', 'sftp',
	'svn', 'svn+ssh', 'ws', 'wss']

	uses_netloc = ['', 'ftp', 'http', 'gopher', 'nntp', 'telnet',
	'imap', 'wais', 'file', 'mms', 'https', 'shttp',
	'snews', 'prospero', 'rtsp', 'rtspu', 'rsync',
	'svn', 'svn+ssh', 'sftp', 'nfs', 'git', 'git+ssh',
	'ws', 'wss']

	uses_params = ['', 'ftp', 'hdl', 'prospero', 'http', 'imap',
	'https', 'shttp', 'rtsp', 'rtspu', 'sip', 'sips',
	'mms', 'sftp', 'tel']

	# These are not actually used anymore, but should stay for backwards
	# compatibility. (They are undocumented, but have a public-looking name.)

	non_hierarchical = ['gopher', 'hdl', 'mailto', 'news',
	'telnet', 'wais', 'imap', 'snews', 'sip', 'sips']

	uses_query = ['', 'http', 'wais', 'imap', 'https', 'shttp', 'mms',
	'gopher', 'rtsp', 'rtspu', 'sip', 'sips']

	uses_fragment = ['', 'ftp', 'hdl', 'http', 'gopher', 'news',
	'nntp', 'wais', 'https', 'shttp', 'snews',
	'file', 'prospero']

urlunsplit for itms-services scheme returns invalid url #104139

Description

Your environment

Linked PRs

Activity

davidism commented on May 3, 2023

davidism commented on May 8, 2023

gpshead commented on May 8, 2023

gpshead commented on May 9, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions